HTTP POST
POST is the HTTP method that was invented to send data to a receiving web application, and it is how most common HTML forms on the web works. It usually sends a chunk of relatively small amounts of data to the receiver.
When the data is sent by a browser after
data have been filled in a form, it will
send it URL encoded, as a serialized
name=value pairs separated with ampersand
symbols (&). You send such data with curl's -d
or --data
option like this:
curl -d 'name=admin&shoesize=12' http://example.com/
When specifying multiple -d
options on the command line, curl will
concatenate them and insert ampersands in
between, so the above example could also be
made like this:
curl -d name=admin -d shoesize=12 http://example.com/
If the amount of data to send is not really fit to put in a mere string on the command line, you can also read it off a file name in standard curl style:
curl -d @filename http://example.com
POSTing with curl's -d option will
make it include a default header that looks
like Content-Type:
application/x-www-form-urlencoded. That's what your typical browser
will use for a plain POST.
Many receivers of POST data do not care about or check the Content-Type header.
If that header is not good enough for you, you should, of course, replace that and instead provide the correct one. Such as if you POST JSON to a server and want to more accurately tell the server about what the content is:
curl -d '{json}' -H 'Content-Type: application/json' https://example.com
When reading from a file, -d
will strip out carriage return and newlines.
Use --data-binary
if you want curl to read and use the given
file in binary exactly as given:
curl --data-binary @filename http://example.com/
Percent-encoding, also known as URL
encoding, is technically a mechanism for
encoding data so that it can appear in URLs.
This encoding is typically used when sending
POSTs with the application/x-www-form-urlencoded
content type, such as the ones curl sends
with --data
and --data-binary
etc.
The command-line options mentioned above all require that you provide properly encoded data, data you need to make sure already exists in the right format. While that gives you a lot of freedom, it is also a bit inconvenient at times.
To help you send data you have not already
encoded, curl offers the --data-urlencode
option. This option offers several different
ways to URL encode the data you give
it.
You use it like --data-urlencode data
in the same style as the other --data
options. To be CGI-compliant, the data
part should begin with a name followed by a
separator and a content specification. The data
part can be passed to curl using one of the
following syntaxes:
-
content: This will make curl URL encode the content and pass that on. Just be careful so that the content does not contain any = or @ symbols, as that will then make the syntax match one of the other cases below! -
=content: This will make curl URL encode the content and pass that on. The initial '=' symbol is not included in the data. -
name=content: This will make curl URL encode the content part and pass that on. Note that the name part is expected to be URL encoded already. -
@filename: This will make curl load data from the given file (including any newlines), URL encode that data and pass it on in the POST. -
[email protected]: This will make curl load data from the given file (including any newlines), URL encode that data and pass it on in the POST. The name part gets an equal sign appended, resulting in name=urlencoded-file-content. Note that the name is expected to be URL encoded already.
As an example, you could POST a name to have it encoded by curl:
curl --data-urlencode "name=John Doe (Junior)" http://example.com
…which would send the following data in the actual request body:
name=John%20Doe%20%28Junior%29
If you store the string John Doe (Junior)
in a file named contents.txt, you can tell curl to send that contents
URL encoded using the field name
'user' like this:
curl --data-urlencode [email protected] http://example.com
In both these examples above, the field
name is not URL encoded but is passed on
as-is. If you want to URL encode the field
name as well, like if you want to pass on a
field name called user name, you can ask curl to encode the entire
string by prefixing it with an equals sign
(that will not get sent):
curl --data-urlencode "=user name=John Doe (Junior)" http://example.com
A little convenience feature that could be
suitable to mention in this context is the -G
or --get
option, which takes all data you have
specified with the different -d
variants and appends that data to the
inputted URL e.g. http://example.com
separated with a '?' and then
makes curl send a GET instead.
This option makes it easy to switch between POSTing and GETing a form, for example.
HTTP has no proper way to stop an ongoing transfer (in any direction) and still maintain the connection. So, if we figure out that the transfer had better stop after the transfer has started, there are only two ways to proceed: cut the connection and pay the price of reestablishing the connection again for the next request, or keep the transfer going and waste bandwidth but be able to reuse the connection next time.
One example of when this can happen is when you send a large file over HTTP, only to discover that the server requires authentication and immediately sends back a 401 response code.
The mitigation that exists to make this
scenario less frequent is to have curl pass
on an extra header, Expect: 100-continue, which gives the server a chance to deny
the request before a lot of data is sent
off. curl sends this Expect: header by
default if the POST it will do is known or
suspected to be larger than just minuscule.
curl also does this for PUT requests.
When a server gets a request with an 100-continue and deems the request fine, it will respond with a 100 response that makes the client continue. If the server does not like the request, it sends back response code for the error it thinks it is.
Unfortunately, lots of servers in the world do not properly support the Expect: header or do not handle it correctly, so curl will only wait 1000 milliseconds for that first response before it will continue anyway.
Those are 1000 wasted milliseconds. You can
then remove the use of Expect: from the
request and avoid the waiting with -H:
curl -H Expect: -d "payload to send" http://example.com
In some situations, curl will inhibit the use of the Expect header if the content it is about to send is small (like below one kilobyte), as having to waste such a small chunk of data is not considered much of a problem.
When talking to a HTTP 1.1 server, you can
tell curl to send the request body without a Content-Length:
header upfront that specifies exactly how
big the POST is. By insisting on curl using
chunked Transfer-Encoding, curl will send
the POST chunked piece by piece in a special
style that also sends the size for each such
chunk as it goes along.
You send a chunked POST with curl like this:
curl -H "Transfer-Encoding: chunked" -d "payload to send" http://example.com
This chapter has explained how sending a
post with -d
is the equivalent of what a browser does
when an HTML form is filled in and
submitted.
Submitting such forms is a common operation with curl; effectively, to have curl fill in a web form in an automated fashion.
If you want to submit a form with curl and make it look as if it has been done with a browser, it is important to provide all the input fields from the form. A common method for web pages is to set a few hidden input fields to the form and have them assigned values directly in the HTML. A successful form submission, of course, also includes those fields and in order to do that automatically you may be forced to first download the HTML page that holds the form, parse it, and extract the hidden field values so that you can send them off with curl.
A common shortcut is to simply fill in the form with your browser and submit it and check in the browser's network development tools exactly what it sent.
A slightly different way is to save the HTML page containing the form, and then edit that HTML page to redirect the 'action=' part of the form to your own server or a test server that just outputs exactly what it gets. Completing that form submission will then show you exactly how a browser sends it.
A third option is, of course, to use a network capture tool such as Wireshark to check exactly what is sent over the wire. If you are working with HTTPS, you cannot see form submissions in clear text on the wire but instead you need to make sure you can have Wireshark extract your TLS private key from your browser. See the Wireshark documentation for details on doing that.
A common mitigation against automated agents or scripts using curl is to have the page with the HTML form use JavaScript to set values of some input fields, usually one of the hidden ones. Often, there's some JavaScript code that executes on page load or when the submit button is pressed which sets a magic value that the server then can verify before it considers the submission to be valid.
You can usually work around that by just reading the JavaScript code and redoing that logic in your script. Using the above mentioned tricks to check exactly what a browser sends is then also a good help.
