1/*************************************************************************** 2 * _ _ ____ _ 3 * Project ___| | | | _ \| | 4 * / __| | | | |_) | | 5 * | (__| |_| | _ <| |___ 6 * \___|\___/|_| \_\_____| 7 * 8 * Copyright (C) 1998 - 2016, Daniel Stenberg, <daniel@haxx.se>, et al. 9 * 10 * This software is licensed as described in the file COPYING, which 11 * you should have received as part of this distribution. The terms 12 * are also available at https://curl.haxx.se/docs/copyright.html. 13 * 14 * You may opt to use, copy, modify, merge, publish, distribute and/or sell 15 * copies of the Software, and permit persons to whom the Software is 16 * furnished to do so, under the terms of the COPYING file. 17 * 18 * This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY 19 * KIND, either express or implied. 20 * 21 ***************************************************************************/ 22#include "tool_setup.h" 23 24#ifdef HAVE_FCNTL_H 25# include <fcntl.h> 26#endif 27 28#ifdef HAVE_UTIME_H 29# include <utime.h> 30#elif defined(HAVE_SYS_UTIME_H) 31# include <sys/utime.h> 32#endif 33 34#ifdef HAVE_LOCALE_H 35# include <locale.h> 36#endif 37 38#ifdef HAVE_NETINET_TCP_H 39# include <netinet/tcp.h> 40#endif 41 42#ifdef __VMS 43# include <fabdef.h> 44#endif 45 46#include "rawstr.h" 47 48#define ENABLE_CURLX_PRINTF 49/* use our own printf() functions */ 50#include "curlx.h" 51 52#include "tool_binmode.h" 53#include "tool_cfgable.h" 54#include "tool_cb_dbg.h" 55#include "tool_cb_hdr.h" 56#include "tool_cb_prg.h" 57#include "tool_cb_rea.h" 58#include "tool_cb_see.h" 59#include "tool_cb_wrt.h" 60#include "tool_dirhie.h" 61#include "tool_doswin.h" 62#include "tool_easysrc.h" 63#include "tool_getparam.h" 64#include "tool_helpers.h" 65#include "tool_homedir.h" 66#include "tool_libinfo.h" 67#include "tool_main.h" 68#include "tool_metalink.h" 69#include "tool_msgs.h" 70#include "tool_operate.h" 71#include "tool_operhlp.h" 72#include "tool_paramhlp.h" 73#include "tool_parsecfg.h" 74#include "tool_setopt.h" 75#include "tool_sleep.h" 76#include "tool_urlglob.h" 77#include "tool_util.h" 78#include "tool_writeenv.h" 79#include "tool_writeout.h" 80#include "tool_xattr.h" 81#include "tool_vms.h" 82#include "tool_help.h" 83#include "tool_hugehelp.h" 84 85#include "memdebug.h" /* keep this as LAST include */ 86 87#ifdef CURLDEBUG 88/* libcurl's debug builds provide an extra function */ 89CURLcode curl_easy_perform_ev(CURL *easy); 90#endif 91 92#define CURLseparator "--_curl_--" 93 94#ifndef O_BINARY 95/* since O_BINARY as used in bitmasks, setting it to zero makes it usable in 96 source code but yet it doesn't ruin anything */ 97# define O_BINARY 0 98#endif 99 100#define CURL_CA_CERT_ERRORMSG1 \ 101 "More details here: https://curl.haxx.se/docs/sslcerts.html\n\n" \ 102 "curl performs SSL certificate verification by default, " \ 103 "using a \"bundle\"\n" \ 104 " of Certificate Authority (CA) public keys (CA certs). If the default\n" \ 105 " bundle file isn't adequate, you can specify an alternate file\n" \ 106 " using the --cacert option.\n" 107 108#define CURL_CA_CERT_ERRORMSG2 \ 109 "If this HTTPS server uses a certificate signed by a CA represented in\n" \ 110 " the bundle, the certificate verification probably failed due to a\n" \ 111 " problem with the certificate (it might be expired, or the name might\n" \ 112 " not match the domain name in the URL).\n" \ 113 "If you'd like to turn off curl's verification of the certificate, use\n" \ 114 " the -k (or --insecure) option.\n" 115 116static bool is_fatal_error(CURLcode code) 117{ 118 switch(code) { 119 /* TODO: Should CURLE_SSL_CACERT be included as critical error ? */ 120 case CURLE_FAILED_INIT: 121 case CURLE_OUT_OF_MEMORY: 122 case CURLE_UNKNOWN_OPTION: 123 case CURLE_FUNCTION_NOT_FOUND: 124 case CURLE_BAD_FUNCTION_ARGUMENT: 125 /* critical error */ 126 return TRUE; 127 default: 128 break; 129 } 130 131 /* no error or not critical */ 132 return FALSE; 133} 134 135#ifdef __VMS 136/* 137 * get_vms_file_size does what it takes to get the real size of the file 138 * 139 * For fixed files, find out the size of the EOF block and adjust. 140 * 141 * For all others, have to read the entire file in, discarding the contents. 142 * Most posted text files will be small, and binary files like zlib archives 143 * and CD/DVD images should be either a STREAM_LF format or a fixed format. 144 * 145 */ 146static curl_off_t vms_realfilesize(const char * name, 147 const struct_stat * stat_buf) 148{ 149 char buffer[8192]; 150 curl_off_t count; 151 int ret_stat; 152 FILE * file; 153 154 /* !checksrc! disable FOPENMODE 1 */ 155 file = fopen(name, "r"); /* VMS */ 156 if(file == NULL) { 157 return 0; 158 } 159 count = 0; 160 ret_stat = 1; 161 while(ret_stat > 0) { 162 ret_stat = fread(buffer, 1, sizeof(buffer), file); 163 if(ret_stat != 0) 164 count += ret_stat; 165 } 166 fclose(file); 167 168 return count; 169} 170 171/* 172 * 173 * VmsSpecialSize checks to see if the stat st_size can be trusted and 174 * if not to call a routine to get the correct size. 175 * 176 */ 177static curl_off_t VmsSpecialSize(const char * name, 178 const struct_stat * stat_buf) 179{ 180 switch(stat_buf->st_fab_rfm) { 181 case FAB$C_VAR: 182 case FAB$C_VFC: 183 return vms_realfilesize(name, stat_buf); 184 break; 185 default: 186 return stat_buf->st_size; 187 } 188} 189#endif /* __VMS */ 190 191static CURLcode operate_do(struct GlobalConfig *global, 192 struct OperationConfig *config) 193{ 194 char errorbuffer[CURL_ERROR_SIZE]; 195 struct ProgressData progressbar; 196 struct getout *urlnode; 197 198 struct HdrCbData hdrcbdata; 199 struct OutStruct heads; 200 201 metalinkfile *mlfile_last = NULL; 202 203 CURL *curl = config->easy; 204 char *httpgetfields = NULL; 205 206 CURLcode result = CURLE_OK; 207 unsigned long li; 208 bool capath_from_env; 209 210 /* Save the values of noprogress and isatty to restore them later on */ 211 bool orig_noprogress = global->noprogress; 212 bool orig_isatty = global->isatty; 213 214 errorbuffer[0] = '\0'; 215 216 /* default headers output stream is stdout */ 217 memset(&hdrcbdata, 0, sizeof(struct HdrCbData)); 218 memset(&heads, 0, sizeof(struct OutStruct)); 219 heads.stream = stdout; 220 heads.config = config; 221 222 /* 223 ** Beyond this point no return'ing from this function allowed. 224 ** Jump to label 'quit_curl' in order to abandon this function 225 ** from outside of nested loops further down below. 226 */ 227 228 /* Check we have a url */ 229 if(!config->url_list || !config->url_list->url) { 230 helpf(global->errors, "no URL specified!\n"); 231 result = CURLE_FAILED_INIT; 232 goto quit_curl; 233 } 234 235 /* On WIN32 we can't set the path to curl-ca-bundle.crt 236 * at compile time. So we look here for the file in two ways: 237 * 1: look at the environment variable CURL_CA_BUNDLE for a path 238 * 2: if #1 isn't found, use the windows API function SearchPath() 239 * to find it along the app's path (includes app's dir and CWD) 240 * 241 * We support the environment variable thing for non-Windows platforms 242 * too. Just for the sake of it. 243 */ 244 capath_from_env = false; 245 if(!config->cacert && 246 !config->capath && 247 !config->insecure_ok) { 248 char *env; 249 env = curlx_getenv("CURL_CA_BUNDLE"); 250 if(env) { 251 config->cacert = strdup(env); 252 if(!config->cacert) { 253 curl_free(env); 254 helpf(global->errors, "out of memory\n"); 255 result = CURLE_OUT_OF_MEMORY; 256 goto quit_curl; 257 } 258 } 259 else { 260 env = curlx_getenv("SSL_CERT_DIR"); 261 if(env) { 262 config->capath = strdup(env); 263 if(!config->capath) { 264 curl_free(env); 265 helpf(global->errors, "out of memory\n"); 266 result = CURLE_OUT_OF_MEMORY; 267 goto quit_curl; 268 } 269 capath_from_env = true; 270 } 271 else { 272 env = curlx_getenv("SSL_CERT_FILE"); 273 if(env) { 274 config->cacert = strdup(env); 275 if(!config->cacert) { 276 curl_free(env); 277 helpf(global->errors, "out of memory\n"); 278 result = CURLE_OUT_OF_MEMORY; 279 goto quit_curl; 280 } 281 } 282 } 283 } 284 285 if(env) 286 curl_free(env); 287#ifdef WIN32 288 else { 289 result = FindWin32CACert(config, "curl-ca-bundle.crt"); 290 if(result) 291 goto quit_curl; 292 } 293#endif 294 } 295 296 if(config->postfields) { 297 if(config->use_httpget) { 298 /* Use the postfields data for a http get */ 299 httpgetfields = strdup(config->postfields); 300 Curl_safefree(config->postfields); 301 if(!httpgetfields) { 302 helpf(global->errors, "out of memory\n"); 303 result = CURLE_OUT_OF_MEMORY; 304 goto quit_curl; 305 } 306 if(SetHTTPrequest(config, 307 (config->no_body?HTTPREQ_HEAD:HTTPREQ_GET), 308 &config->httpreq)) { 309 result = CURLE_FAILED_INIT; 310 goto quit_curl; 311 } 312 } 313 else { 314 if(SetHTTPrequest(config, HTTPREQ_SIMPLEPOST, &config->httpreq)) { 315 result = CURLE_FAILED_INIT; 316 goto quit_curl; 317 } 318 } 319 } 320 321 /* Single header file for all URLs */ 322 if(config->headerfile) { 323 /* open file for output: */ 324 if(!curlx_strequal(config->headerfile, "-")) { 325 FILE *newfile = fopen(config->headerfile, "wb"); 326 if(!newfile) { 327 warnf(config->global, "Failed to open %s\n", config->headerfile); 328 result = CURLE_WRITE_ERROR; 329 goto quit_curl; 330 } 331 else { 332 heads.filename = config->headerfile; 333 heads.s_isreg = TRUE; 334 heads.fopened = TRUE; 335 heads.stream = newfile; 336 } 337 } 338 else { 339 /* always use binary mode for protocol header output */ 340 set_binmode(heads.stream); 341 } 342 } 343 344 /* 345 ** Nested loops start here. 346 */ 347 348 /* loop through the list of given URLs */ 349 350 for(urlnode = config->url_list; urlnode; urlnode = urlnode->next) { 351 352 unsigned long up; /* upload file counter within a single upload glob */ 353 char *infiles; /* might be a glob pattern */ 354 char *outfiles; 355 unsigned long infilenum; 356 URLGlob *inglob; 357 358 int metalink = 0; /* nonzero for metalink download. */ 359 metalinkfile *mlfile; 360 metalink_resource *mlres; 361 362 outfiles = NULL; 363 infilenum = 1; 364 inglob = NULL; 365 366 if(urlnode->flags & GETOUT_METALINK) { 367 metalink = 1; 368 if(mlfile_last == NULL) { 369 mlfile_last = config->metalinkfile_list; 370 } 371 mlfile = mlfile_last; 372 mlfile_last = mlfile_last->next; 373 mlres = mlfile->resource; 374 } 375 else { 376 mlfile = NULL; 377 mlres = NULL; 378 } 379 380 /* urlnode->url is the full URL (it might be NULL) */ 381 382 if(!urlnode->url) { 383 /* This node has no URL. Free node data without destroying the 384 node itself nor modifying next pointer and continue to next */ 385 Curl_safefree(urlnode->outfile); 386 Curl_safefree(urlnode->infile); 387 urlnode->flags = 0; 388 continue; /* next URL please */ 389 } 390 391 /* save outfile pattern before expansion */ 392 if(urlnode->outfile) { 393 outfiles = strdup(urlnode->outfile); 394 if(!outfiles) { 395 helpf(global->errors, "out of memory\n"); 396 result = CURLE_OUT_OF_MEMORY; 397 break; 398 } 399 } 400 401 infiles = urlnode->infile; 402 403 if(!config->globoff && infiles) { 404 /* Unless explicitly shut off */ 405 result = glob_url(&inglob, infiles, &infilenum, 406 global->showerror?global->errors:NULL); 407 if(result) { 408 Curl_safefree(outfiles); 409 break; 410 } 411 } 412 413 /* Here's the loop for uploading multiple files within the same 414 single globbed string. If no upload, we enter the loop once anyway. */ 415 for(up = 0 ; up < infilenum; up++) { 416 417 char *uploadfile; /* a single file, never a glob */ 418 int separator; 419 URLGlob *urls; 420 unsigned long urlnum; 421 422 uploadfile = NULL; 423 urls = NULL; 424 urlnum = 0; 425 426 if(!up && !infiles) 427 Curl_nop_stmt; 428 else { 429 if(inglob) { 430 result = glob_next_url(&uploadfile, inglob); 431 if(result == CURLE_OUT_OF_MEMORY) 432 helpf(global->errors, "out of memory\n"); 433 } 434 else if(!up) { 435 uploadfile = strdup(infiles); 436 if(!uploadfile) { 437 helpf(global->errors, "out of memory\n"); 438 result = CURLE_OUT_OF_MEMORY; 439 } 440 } 441 else 442 uploadfile = NULL; 443 if(!uploadfile) 444 break; 445 } 446 447 if(metalink) { 448 /* For Metalink download, we don't use glob. Instead we use 449 the number of resources as urlnum. */ 450 urlnum = count_next_metalink_resource(mlfile); 451 } 452 else 453 if(!config->globoff) { 454 /* Unless explicitly shut off, we expand '{...}' and '[...]' 455 expressions and return total number of URLs in pattern set */ 456 result = glob_url(&urls, urlnode->url, &urlnum, 457 global->showerror?global->errors:NULL); 458 if(result) { 459 Curl_safefree(uploadfile); 460 break; 461 } 462 } 463 else 464 urlnum = 1; /* without globbing, this is a single URL */ 465 466 /* if multiple files extracted to stdout, insert separators! */ 467 separator= ((!outfiles || curlx_strequal(outfiles, "-")) && urlnum > 1); 468 469 /* Here's looping around each globbed URL */ 470 for(li = 0 ; li < urlnum; li++) { 471 472 int infd; 473 bool infdopen; 474 char *outfile; 475 struct OutStruct outs; 476 struct InStruct input; 477 struct timeval retrystart; 478 curl_off_t uploadfilesize; 479 long retry_numretries; 480 long retry_sleep_default; 481 long retry_sleep; 482 char *this_url = NULL; 483 int metalink_next_res = 0; 484 485 outfile = NULL; 486 infdopen = FALSE; 487 infd = STDIN_FILENO; 488 uploadfilesize = -1; /* -1 means unknown */ 489 490 /* default output stream is stdout */ 491 memset(&outs, 0, sizeof(struct OutStruct)); 492 outs.stream = stdout; 493 outs.config = config; 494 495 if(metalink) { 496 /* For Metalink download, use name in Metalink file as 497 filename. */ 498 outfile = strdup(mlfile->filename); 499 if(!outfile) { 500 result = CURLE_OUT_OF_MEMORY; 501 goto show_error; 502 } 503 this_url = strdup(mlres->url); 504 if(!this_url) { 505 result = CURLE_OUT_OF_MEMORY; 506 goto show_error; 507 } 508 } 509 else { 510 if(urls) { 511 result = glob_next_url(&this_url, urls); 512 if(result) 513 goto show_error; 514 } 515 else if(!li) { 516 this_url = strdup(urlnode->url); 517 if(!this_url) { 518 result = CURLE_OUT_OF_MEMORY; 519 goto show_error; 520 } 521 } 522 else 523 this_url = NULL; 524 if(!this_url) 525 break; 526 527 if(outfiles) { 528 outfile = strdup(outfiles); 529 if(!outfile) { 530 result = CURLE_OUT_OF_MEMORY; 531 goto show_error; 532 } 533 } 534 } 535 536 if(((urlnode->flags&GETOUT_USEREMOTE) || 537 (outfile && !curlx_strequal("-", outfile))) && 538 (metalink || !config->use_metalink)) { 539 540 /* 541 * We have specified a file name to store the result in, or we have 542 * decided we want to use the remote file name. 543 */ 544 545 if(!outfile) { 546 /* extract the file name from the URL */ 547 result = get_url_file_name(&outfile, this_url); 548 if(result) 549 goto show_error; 550 if(!*outfile && !config->content_disposition) { 551 helpf(global->errors, "Remote file name has no length!\n"); 552 result = CURLE_WRITE_ERROR; 553 goto quit_urls; 554 } 555 } 556 else if(urls) { 557 /* fill '#1' ... '#9' terms from URL pattern */ 558 char *storefile = outfile; 559 result = glob_match_url(&outfile, storefile, urls); 560 Curl_safefree(storefile); 561 if(result) { 562 /* bad globbing */ 563 warnf(config->global, "bad output glob!\n"); 564 goto quit_urls; 565 } 566 } 567 568 /* Create the directory hierarchy, if not pre-existent to a multiple 569 file output call */ 570 571 if(config->create_dirs || metalink) { 572 result = create_dir_hierarchy(outfile, global->errors); 573 /* create_dir_hierarchy shows error upon CURLE_WRITE_ERROR */ 574 if(result == CURLE_WRITE_ERROR) 575 goto quit_urls; 576 if(result) { 577 goto show_error; 578 } 579 } 580 581 if((urlnode->flags & GETOUT_USEREMOTE) 582 && config->content_disposition) { 583 /* Our header callback MIGHT set the filename */ 584 DEBUGASSERT(!outs.filename); 585 } 586 587 if(config->resume_from_current) { 588 /* We're told to continue from where we are now. Get the size 589 of the file as it is now and open it for append instead */ 590 struct_stat fileinfo; 591 /* VMS -- Danger, the filesize is only valid for stream files */ 592 if(0 == stat(outfile, &fileinfo)) 593 /* set offset to current file size: */ 594 config->resume_from = fileinfo.st_size; 595 else 596 /* let offset be 0 */ 597 config->resume_from = 0; 598 } 599 600 if(config->resume_from) { 601#ifdef __VMS 602 /* open file for output, forcing VMS output format into stream 603 mode which is needed for stat() call above to always work. */ 604 FILE *file = fopen(outfile, config->resume_from?"ab":"wb", 605 "ctx=stm", "rfm=stmlf", "rat=cr", "mrs=0"); 606#else 607 /* open file for output: */ 608 FILE *file = fopen(outfile, config->resume_from?"ab":"wb"); 609#endif 610 if(!file) { 611 helpf(global->errors, "Can't open '%s'!\n", outfile); 612 result = CURLE_WRITE_ERROR; 613 goto quit_urls; 614 } 615 outs.fopened = TRUE; 616 outs.stream = file; 617 outs.init = config->resume_from; 618 } 619 else { 620 outs.stream = NULL; /* open when needed */ 621 } 622 outs.filename = outfile; 623 outs.s_isreg = TRUE; 624 } 625 626 if(uploadfile && !stdin_upload(uploadfile)) { 627 /* 628 * We have specified a file to upload and it isn't "-". 629 */ 630 struct_stat fileinfo; 631 632 this_url = add_file_name_to_url(curl, this_url, uploadfile); 633 if(!this_url) { 634 result = CURLE_OUT_OF_MEMORY; 635 goto show_error; 636 } 637 /* VMS Note: 638 * 639 * Reading binary from files can be a problem... Only FIXED, VAR 640 * etc WITHOUT implied CC will work Others need a \n appended to a 641 * line 642 * 643 * - Stat gives a size but this is UNRELIABLE in VMS As a f.e. a 644 * fixed file with implied CC needs to have a byte added for every 645 * record processed, this can by derived from Filesize & recordsize 646 * for VARiable record files the records need to be counted! for 647 * every record add 1 for linefeed and subtract 2 for the record 648 * header for VARIABLE header files only the bare record data needs 649 * to be considered with one appended if implied CC 650 */ 651#ifdef __VMS 652 /* Calculate the real upload site for VMS */ 653 infd = -1; 654 if(stat(uploadfile, &fileinfo) == 0) { 655 fileinfo.st_size = VmsSpecialSize(uploadfile, &fileinfo); 656 switch (fileinfo.st_fab_rfm) { 657 case FAB$C_VAR: 658 case FAB$C_VFC: 659 case FAB$C_STMCR: 660 infd = open(uploadfile, O_RDONLY | O_BINARY); 661 break; 662 default: 663 infd = open(uploadfile, O_RDONLY | O_BINARY, 664 "rfm=stmlf", "ctx=stm"); 665 } 666 } 667 if(infd == -1) 668#else 669 infd = open(uploadfile, O_RDONLY | O_BINARY); 670 if((infd == -1) || fstat(infd, &fileinfo)) 671#endif 672 { 673 helpf(global->errors, "Can't open '%s'!\n", uploadfile); 674 if(infd != -1) { 675 close(infd); 676 infd = STDIN_FILENO; 677 } 678 result = CURLE_READ_ERROR; 679 goto quit_urls; 680 } 681 infdopen = TRUE; 682 683 /* we ignore file size for char/block devices, sockets, etc. */ 684 if(S_ISREG(fileinfo.st_mode)) 685 uploadfilesize = fileinfo.st_size; 686 687 } 688 else if(uploadfile && stdin_upload(uploadfile)) { 689 /* count to see if there are more than one auth bit set 690 in the authtype field */ 691 int authbits = 0; 692 int bitcheck = 0; 693 while(bitcheck < 32) { 694 if(config->authtype & (1UL << bitcheck++)) { 695 authbits++; 696 if(authbits > 1) { 697 /* more than one, we're done! */ 698 break; 699 } 700 } 701 } 702 703 /* 704 * If the user has also selected --anyauth or --proxy-anyauth 705 * we should warn him/her. 706 */ 707 if(config->proxyanyauth || (authbits>1)) { 708 warnf(config->global, 709 "Using --anyauth or --proxy-anyauth with upload from stdin" 710 " involves a big risk of it not working. Use a temporary" 711 " file or a fixed auth type instead!\n"); 712 } 713 714 DEBUGASSERT(infdopen == FALSE); 715 DEBUGASSERT(infd == STDIN_FILENO); 716 717 set_binmode(stdin); 718 if(curlx_strequal(uploadfile, ".")) { 719 if(curlx_nonblock((curl_socket_t)infd, TRUE) < 0) 720 warnf(config->global, 721 "fcntl failed on fd=%d: %s\n", infd, strerror(errno)); 722 } 723 } 724 725 if(uploadfile && config->resume_from_current) 726 config->resume_from = -1; /* -1 will then force get-it-yourself */ 727 728 if(output_expected(this_url, uploadfile) && outs.stream && 729 isatty(fileno(outs.stream))) 730 /* we send the output to a tty, therefore we switch off the progress 731 meter */ 732 global->noprogress = global->isatty = TRUE; 733 else { 734 /* progress meter is per download, so restore config 735 values */ 736 global->noprogress = orig_noprogress; 737 global->isatty = orig_isatty; 738 } 739 740 if(urlnum > 1 && !global->mute) { 741 fprintf(global->errors, "\n[%lu/%lu]: %s --> %s\n", 742 li+1, urlnum, this_url, outfile ? outfile : "<stdout>"); 743 if(separator) 744 printf("%s%s\n", CURLseparator, this_url); 745 } 746 if(httpgetfields) { 747 char *urlbuffer; 748 /* Find out whether the url contains a file name */ 749 const char *pc = strstr(this_url, "://"); 750 char sep = '?'; 751 if(pc) 752 pc += 3; 753 else 754 pc = this_url; 755 756 pc = strrchr(pc, '/'); /* check for a slash */ 757 758 if(pc) { 759 /* there is a slash present in the URL */ 760 761 if(strchr(pc, '?')) 762 /* Ouch, there's already a question mark in the URL string, we 763 then append the data with an ampersand separator instead! */ 764 sep='&'; 765 } 766 /* 767 * Then append ? followed by the get fields to the url. 768 */ 769 if(pc) 770 urlbuffer = aprintf("%s%c%s", this_url, sep, httpgetfields); 771 else 772 /* Append / before the ? to create a well-formed url 773 if the url contains a hostname only 774 */ 775 urlbuffer = aprintf("%s/?%s", this_url, httpgetfields); 776 777 if(!urlbuffer) { 778 result = CURLE_OUT_OF_MEMORY; 779 goto show_error; 780 } 781 782 Curl_safefree(this_url); /* free previous URL */ 783 this_url = urlbuffer; /* use our new URL instead! */ 784 } 785 786 if(!global->errors) 787 global->errors = stderr; 788 789 if((!outfile || !strcmp(outfile, "-")) && !config->use_ascii) { 790 /* We get the output to stdout and we have not got the ASCII/text 791 flag, then set stdout to be binary */ 792 set_binmode(stdout); 793 } 794 795 if(config->tcp_nodelay) 796 my_setopt(curl, CURLOPT_TCP_NODELAY, 1L); 797 798 if(config->tcp_fastopen) 799 my_setopt(curl, CURLOPT_TCP_FASTOPEN, 1L); 800 801 /* where to store */ 802 my_setopt(curl, CURLOPT_WRITEDATA, &outs); 803 if(metalink || !config->use_metalink) 804 /* what call to write */ 805 my_setopt(curl, CURLOPT_WRITEFUNCTION, tool_write_cb); 806#ifdef USE_METALINK 807 else 808 /* Set Metalink specific write callback function to parse 809 XML data progressively. */ 810 my_setopt(curl, CURLOPT_WRITEFUNCTION, metalink_write_cb); 811#endif /* USE_METALINK */ 812 813 /* for uploads */ 814 input.fd = infd; 815 input.config = config; 816 /* Note that if CURLOPT_READFUNCTION is fread (the default), then 817 * lib/telnet.c will Curl_poll() on the input file descriptor 818 * rather then calling the READFUNCTION at regular intervals. 819 * The circumstances in which it is preferable to enable this 820 * behaviour, by omitting to set the READFUNCTION & READDATA options, 821 * have not been determined. 822 */ 823 my_setopt(curl, CURLOPT_READDATA, &input); 824 /* what call to read */ 825 my_setopt(curl, CURLOPT_READFUNCTION, tool_read_cb); 826 827 /* in 7.18.0, the CURLOPT_SEEKFUNCTION/DATA pair is taking over what 828 CURLOPT_IOCTLFUNCTION/DATA pair previously provided for seeking */ 829 my_setopt(curl, CURLOPT_SEEKDATA, &input); 830 my_setopt(curl, CURLOPT_SEEKFUNCTION, tool_seek_cb); 831 832 if(config->recvpersecond) 833 /* tell libcurl to use a smaller sized buffer as it allows us to 834 make better sleeps! 7.9.9 stuff! */ 835 my_setopt(curl, CURLOPT_BUFFERSIZE, (long)config->recvpersecond); 836 837 /* size of uploaded file: */ 838 if(uploadfilesize != -1) 839 my_setopt(curl, CURLOPT_INFILESIZE_LARGE, uploadfilesize); 840 my_setopt_str(curl, CURLOPT_URL, this_url); /* what to fetch */ 841 my_setopt(curl, CURLOPT_NOPROGRESS, global->noprogress?1L:0L); 842 if(config->no_body) { 843 my_setopt(curl, CURLOPT_NOBODY, 1L); 844 my_setopt(curl, CURLOPT_HEADER, 1L); 845 } 846 /* If --metalink is used, we ignore --include (headers in 847 output) option because mixing headers to the body will 848 confuse XML parser and/or hash check will fail. */ 849 else if(!config->use_metalink) 850 my_setopt(curl, CURLOPT_HEADER, config->include_headers?1L:0L); 851 852 if(config->oauth_bearer) 853 my_setopt_str(curl, CURLOPT_XOAUTH2_BEARER, config->oauth_bearer); 854 855#if !defined(CURL_DISABLE_PROXY) 856 { 857 /* TODO: Make this a run-time check instead of compile-time one. */ 858 859 my_setopt_str(curl, CURLOPT_PROXY, config->proxy); 860 my_setopt_str(curl, CURLOPT_PROXYUSERPWD, config->proxyuserpwd); 861 862 /* new in libcurl 7.3 */ 863 my_setopt(curl, CURLOPT_HTTPPROXYTUNNEL, config->proxytunnel?1L:0L); 864 865 /* new in libcurl 7.5 */ 866 if(config->proxy) 867 my_setopt_enum(curl, CURLOPT_PROXYTYPE, (long)config->proxyver); 868 869 /* new in libcurl 7.10 */ 870 if(config->socksproxy) { 871 my_setopt_str(curl, CURLOPT_PROXY, config->socksproxy); 872 my_setopt_enum(curl, CURLOPT_PROXYTYPE, (long)config->socksver); 873 } 874 875 /* new in libcurl 7.10.6 */ 876 if(config->proxyanyauth) 877 my_setopt_bitmask(curl, CURLOPT_PROXYAUTH, 878 (long)CURLAUTH_ANY); 879 else if(config->proxynegotiate) 880 my_setopt_bitmask(curl, CURLOPT_PROXYAUTH, 881 (long)CURLAUTH_GSSNEGOTIATE); 882 else if(config->proxyntlm) 883 my_setopt_bitmask(curl, CURLOPT_PROXYAUTH, 884 (long)CURLAUTH_NTLM); 885 else if(config->proxydigest) 886 my_setopt_bitmask(curl, CURLOPT_PROXYAUTH, 887 (long)CURLAUTH_DIGEST); 888 else if(config->proxybasic) 889 my_setopt_bitmask(curl, CURLOPT_PROXYAUTH, 890 (long)CURLAUTH_BASIC); 891 892 /* new in libcurl 7.19.4 */ 893 my_setopt_str(curl, CURLOPT_NOPROXY, config->noproxy); 894 } 895#endif 896 897 my_setopt(curl, CURLOPT_FAILONERROR, config->failonerror?1L:0L); 898 my_setopt(curl, CURLOPT_UPLOAD, uploadfile?1L:0L); 899 my_setopt(curl, CURLOPT_DIRLISTONLY, config->dirlistonly?1L:0L); 900 my_setopt(curl, CURLOPT_APPEND, config->ftp_append?1L:0L); 901 902 if(config->netrc_opt) 903 my_setopt_enum(curl, CURLOPT_NETRC, (long)CURL_NETRC_OPTIONAL); 904 else if(config->netrc || config->netrc_file) 905 my_setopt_enum(curl, CURLOPT_NETRC, (long)CURL_NETRC_REQUIRED); 906 else 907 my_setopt_enum(curl, CURLOPT_NETRC, (long)CURL_NETRC_IGNORED); 908 909 if(config->netrc_file) 910 my_setopt_str(curl, CURLOPT_NETRC_FILE, config->netrc_file); 911 912 my_setopt(curl, CURLOPT_TRANSFERTEXT, config->use_ascii?1L:0L); 913 if(config->login_options) 914 my_setopt_str(curl, CURLOPT_LOGIN_OPTIONS, config->login_options); 915 my_setopt_str(curl, CURLOPT_USERPWD, config->userpwd); 916 my_setopt_str(curl, CURLOPT_RANGE, config->range); 917 my_setopt(curl, CURLOPT_ERRORBUFFER, errorbuffer); 918 my_setopt(curl, CURLOPT_TIMEOUT_MS, (long)(config->timeout * 1000)); 919 920 if(built_in_protos & CURLPROTO_HTTP) { 921 922 long postRedir = 0; 923 924 my_setopt(curl, CURLOPT_FOLLOWLOCATION, 925 config->followlocation?1L:0L); 926 my_setopt(curl, CURLOPT_UNRESTRICTED_AUTH, 927 config->unrestricted_auth?1L:0L); 928 929 switch(config->httpreq) { 930 case HTTPREQ_SIMPLEPOST: 931 my_setopt_str(curl, CURLOPT_POSTFIELDS, 932 config->postfields); 933 my_setopt(curl, CURLOPT_POSTFIELDSIZE_LARGE, 934 config->postfieldsize); 935 break; 936 case HTTPREQ_FORMPOST: 937 my_setopt_httppost(curl, CURLOPT_HTTPPOST, config->httppost); 938 break; 939 default: 940 break; 941 } 942 943 my_setopt_str(curl, CURLOPT_REFERER, config->referer); 944 my_setopt(curl, CURLOPT_AUTOREFERER, config->autoreferer?1L:0L); 945 my_setopt_str(curl, CURLOPT_USERAGENT, config->useragent); 946 my_setopt_slist(curl, CURLOPT_HTTPHEADER, config->headers); 947 948 /* new in libcurl 7.36.0 */ 949 if(config->proxyheaders) { 950 my_setopt_slist(curl, CURLOPT_PROXYHEADER, config->proxyheaders); 951 my_setopt(curl, CURLOPT_HEADEROPT, CURLHEADER_SEPARATE); 952 } 953 954 /* new in libcurl 7.5 */ 955 my_setopt(curl, CURLOPT_MAXREDIRS, config->maxredirs); 956 957 if(config->httpversion) 958 my_setopt_enum(curl, CURLOPT_HTTP_VERSION, config->httpversion); 959 else if(curlinfo->features & CURL_VERSION_HTTP2) { 960 my_setopt_enum(curl, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_2TLS); 961 } 962 963 /* new in libcurl 7.10.6 (default is Basic) */ 964 if(config->authtype) 965 my_setopt_bitmask(curl, CURLOPT_HTTPAUTH, (long)config->authtype); 966 967 /* curl 7.19.1 (the 301 version existed in 7.18.2), 968 303 was added in 7.26.0 */ 969 if(config->post301) 970 postRedir |= CURL_REDIR_POST_301; 971 if(config->post302) 972 postRedir |= CURL_REDIR_POST_302; 973 if(config->post303) 974 postRedir |= CURL_REDIR_POST_303; 975 my_setopt(curl, CURLOPT_POSTREDIR, postRedir); 976 977 /* new in libcurl 7.21.6 */ 978 if(config->encoding) 979 my_setopt_str(curl, CURLOPT_ACCEPT_ENCODING, ""); 980 981 /* new in libcurl 7.21.6 */ 982 if(config->tr_encoding) 983 my_setopt(curl, CURLOPT_TRANSFER_ENCODING, 1L); 984 985 } /* (built_in_protos & CURLPROTO_HTTP) */ 986 987 my_setopt_str(curl, CURLOPT_FTPPORT, config->ftpport); 988 my_setopt(curl, CURLOPT_LOW_SPEED_LIMIT, 989 config->low_speed_limit); 990 my_setopt(curl, CURLOPT_LOW_SPEED_TIME, config->low_speed_time); 991 my_setopt(curl, CURLOPT_MAX_SEND_SPEED_LARGE, 992 config->sendpersecond); 993 my_setopt(curl, CURLOPT_MAX_RECV_SPEED_LARGE, 994 config->recvpersecond); 995 996 if(config->use_resume) 997 my_setopt(curl, CURLOPT_RESUME_FROM_LARGE, config->resume_from); 998 else 999 my_setopt(curl, CURLOPT_RESUME_FROM_LARGE, CURL_OFF_T_C(0)); 1000 1001 my_setopt_str(curl, CURLOPT_KEYPASSWD, config->key_passwd); 1002 1003 if(built_in_protos & (CURLPROTO_SCP|CURLPROTO_SFTP)) { 1004 1005 /* SSH and SSL private key uses same command-line option */ 1006 /* new in libcurl 7.16.1 */ 1007 my_setopt_str(curl, CURLOPT_SSH_PRIVATE_KEYFILE, config->key); 1008 /* new in libcurl 7.16.1 */ 1009 my_setopt_str(curl, CURLOPT_SSH_PUBLIC_KEYFILE, config->pubkey); 1010 1011 /* new in libcurl 7.17.1: SSH host key md5 checking allows us 1012 to fail if we are not talking to who we think we should */ 1013 my_setopt_str(curl, CURLOPT_SSH_HOST_PUBLIC_KEY_MD5, 1014 config->hostpubmd5); 1015 } 1016 1017 if(config->cacert) 1018 my_setopt_str(curl, CURLOPT_CAINFO, config->cacert); 1019 if(config->capath) { 1020 result = res_setopt_str(curl, CURLOPT_CAPATH, config->capath); 1021 if(result == CURLE_NOT_BUILT_IN) { 1022 warnf(config->global, "ignoring %s, not supported by libcurl\n", 1023 capath_from_env? 1024 "SSL_CERT_DIR environment variable":"--capath"); 1025 } 1026 else if(result) 1027 goto show_error; 1028 } 1029 if(config->crlfile) 1030 my_setopt_str(curl, CURLOPT_CRLFILE, config->crlfile); 1031 1032 if(config->pinnedpubkey) 1033 my_setopt_str(curl, CURLOPT_PINNEDPUBLICKEY, config->pinnedpubkey); 1034 1035 if(curlinfo->features & CURL_VERSION_SSL) { 1036 my_setopt_str(curl, CURLOPT_SSLCERT, config->cert); 1037 my_setopt_str(curl, CURLOPT_SSLCERTTYPE, config->cert_type); 1038 my_setopt_str(curl, CURLOPT_SSLKEY, config->key); 1039 my_setopt_str(curl, CURLOPT_SSLKEYTYPE, config->key_type); 1040 1041 if(config->insecure_ok) { 1042 my_setopt(curl, CURLOPT_SSL_VERIFYPEER, 0L); 1043 my_setopt(curl, CURLOPT_SSL_VERIFYHOST, 0L); 1044 } 1045 else { 1046 my_setopt(curl, CURLOPT_SSL_VERIFYPEER, 1L); 1047 /* libcurl default is strict verifyhost -> 2L */ 1048 /* my_setopt(curl, CURLOPT_SSL_VERIFYHOST, 2L); */ 1049 } 1050 1051 if(config->verifystatus) 1052 my_setopt(curl, CURLOPT_SSL_VERIFYSTATUS, 1L); 1053 1054 if(config->falsestart) 1055 my_setopt(curl, CURLOPT_SSL_FALSESTART, 1L); 1056 1057 my_setopt_enum(curl, CURLOPT_SSLVERSION, config->ssl_version); 1058 } 1059 if(config->path_as_is) 1060 my_setopt(curl, CURLOPT_PATH_AS_IS, 1L); 1061 1062 if(built_in_protos & (CURLPROTO_SCP|CURLPROTO_SFTP)) { 1063 if(!config->insecure_ok) { 1064 char *home; 1065 char *file; 1066 result = CURLE_OUT_OF_MEMORY; 1067 home = homedir(); 1068 if(home) { 1069 file = aprintf("%s/%sssh/known_hosts", home, DOT_CHAR); 1070 if(file) { 1071 /* new in curl 7.19.6 */ 1072 result = res_setopt_str(curl, CURLOPT_SSH_KNOWNHOSTS, file); 1073 curl_free(file); 1074 if(result == CURLE_UNKNOWN_OPTION) 1075 /* libssh2 version older than 1.1.1 */ 1076 result = CURLE_OK; 1077 } 1078 Curl_safefree(home); 1079 } 1080 if(result) 1081 goto show_error; 1082 } 1083 } 1084 1085 if(config->no_body || config->remote_time) { 1086 /* no body or use remote time */ 1087 my_setopt(curl, CURLOPT_FILETIME, 1L); 1088 } 1089 1090 my_setopt(curl, CURLOPT_CRLF, config->crlf?1L:0L); 1091 my_setopt_slist(curl, CURLOPT_QUOTE, config->quote); 1092 my_setopt_slist(curl, CURLOPT_POSTQUOTE, config->postquote); 1093 my_setopt_slist(curl, CURLOPT_PREQUOTE, config->prequote); 1094 1095#if !defined(CURL_DISABLE_HTTP) && !defined(CURL_DISABLE_COOKIES) 1096 if(config->cookie) 1097 my_setopt_str(curl, CURLOPT_COOKIE, config->cookie); 1098 1099 if(config->cookiefile) 1100 my_setopt_str(curl, CURLOPT_COOKIEFILE, config->cookiefile); 1101 1102 /* new in libcurl 7.9 */ 1103 if(config->cookiejar) 1104 my_setopt_str(curl, CURLOPT_COOKIEJAR, config->cookiejar); 1105 1106 /* new in libcurl 7.9.7 */ 1107 my_setopt(curl, CURLOPT_COOKIESESSION, config->cookiesession?1L:0L); 1108#else 1109 if(config->cookie || config->cookiefile || config->cookiejar) { 1110 warnf(config->global, "cookie option(s) used even though cookie " 1111 "support is disabled!\n"); 1112 return CURLE_NOT_BUILT_IN; 1113 } 1114#endif 1115 1116 my_setopt_enum(curl, CURLOPT_TIMECONDITION, (long)config->timecond); 1117 my_setopt(curl, CURLOPT_TIMEVALUE, (long)config->condtime); 1118 my_setopt_str(curl, CURLOPT_CUSTOMREQUEST, config->customrequest); 1119 customrequest_helper(config, config->httpreq, config->customrequest); 1120 my_setopt(curl, CURLOPT_STDERR, global->errors); 1121 1122 /* three new ones in libcurl 7.3: */ 1123 my_setopt_str(curl, CURLOPT_INTERFACE, config->iface); 1124 my_setopt_str(curl, CURLOPT_KRBLEVEL, config->krblevel); 1125 1126 progressbarinit(&progressbar, config); 1127 if((global->progressmode == CURL_PROGRESS_BAR) && 1128 !global->noprogress && !global->mute) { 1129 /* we want the alternative style, then we have to implement it 1130 ourselves! */ 1131 my_setopt(curl, CURLOPT_XFERINFOFUNCTION, tool_progress_cb); 1132 my_setopt(curl, CURLOPT_XFERINFODATA, &progressbar); 1133 } 1134 1135 /* new in libcurl 7.24.0: */ 1136 if(config->dns_servers) 1137 my_setopt_str(curl, CURLOPT_DNS_SERVERS, config->dns_servers); 1138 1139 /* new in libcurl 7.33.0: */ 1140 if(config->dns_interface) 1141 my_setopt_str(curl, CURLOPT_DNS_INTERFACE, config->dns_interface); 1142 if(config->dns_ipv4_addr) 1143 my_setopt_str(curl, CURLOPT_DNS_LOCAL_IP4, config->dns_ipv4_addr); 1144 if(config->dns_ipv6_addr) 1145 my_setopt_str(curl, CURLOPT_DNS_LOCAL_IP6, config->dns_ipv6_addr); 1146 1147 /* new in libcurl 7.6.2: */ 1148 my_setopt_slist(curl, CURLOPT_TELNETOPTIONS, config->telnet_options); 1149 1150 /* new in libcurl 7.7: */ 1151 my_setopt_str(curl, CURLOPT_RANDOM_FILE, config->random_file); 1152 my_setopt_str(curl, CURLOPT_EGDSOCKET, config->egd_file); 1153 my_setopt(curl, CURLOPT_CONNECTTIMEOUT_MS, 1154 (long)(config->connecttimeout * 1000)); 1155 1156 if(config->cipher_list) 1157 my_setopt_str(curl, CURLOPT_SSL_CIPHER_LIST, config->cipher_list); 1158 1159 /* new in libcurl 7.9.2: */ 1160 if(config->disable_epsv) 1161 /* disable it */ 1162 my_setopt(curl, CURLOPT_FTP_USE_EPSV, 0L); 1163 1164 /* new in libcurl 7.10.5 */ 1165 if(config->disable_eprt) 1166 /* disable it */ 1167 my_setopt(curl, CURLOPT_FTP_USE_EPRT, 0L); 1168 1169 if(global->tracetype != TRACE_NONE) { 1170 my_setopt(curl, CURLOPT_DEBUGFUNCTION, tool_debug_cb); 1171 my_setopt(curl, CURLOPT_DEBUGDATA, config); 1172 my_setopt(curl, CURLOPT_VERBOSE, 1L); 1173 } 1174 1175 /* new in curl 7.9.3 */ 1176 if(config->engine) { 1177 result = res_setopt_str(curl, CURLOPT_SSLENGINE, config->engine); 1178 if(result) 1179 goto show_error; 1180 my_setopt(curl, CURLOPT_SSLENGINE_DEFAULT, 1L); 1181 } 1182 1183 /* new in curl 7.10.7, extended in 7.19.4. Modified to use 1184 CREATE_DIR_RETRY in 7.49.0 */ 1185 my_setopt(curl, CURLOPT_FTP_CREATE_MISSING_DIRS, 1186 (long)(config->ftp_create_dirs? 1187 CURLFTP_CREATE_DIR_RETRY: 1188 CURLFTP_CREATE_DIR_NONE)); 1189 1190 /* new in curl 7.10.8 */ 1191 if(config->max_filesize) 1192 my_setopt(curl, CURLOPT_MAXFILESIZE_LARGE, 1193 config->max_filesize); 1194 1195 if(4 == config->ip_version) 1196 my_setopt(curl, CURLOPT_IPRESOLVE, CURL_IPRESOLVE_V4); 1197 else if(6 == config->ip_version) 1198 my_setopt(curl, CURLOPT_IPRESOLVE, CURL_IPRESOLVE_V6); 1199 else 1200 my_setopt(curl, CURLOPT_IPRESOLVE, CURL_IPRESOLVE_WHATEVER); 1201 1202 /* new in curl 7.15.5 */ 1203 if(config->ftp_ssl_reqd) 1204 my_setopt_enum(curl, CURLOPT_USE_SSL, (long)CURLUSESSL_ALL); 1205 1206 /* new in curl 7.11.0 */ 1207 else if(config->ftp_ssl) 1208 my_setopt_enum(curl, CURLOPT_USE_SSL, (long)CURLUSESSL_TRY); 1209 1210 /* new in curl 7.16.0 */ 1211 else if(config->ftp_ssl_control) 1212 my_setopt_enum(curl, CURLOPT_USE_SSL, (long)CURLUSESSL_CONTROL); 1213 1214 /* new in curl 7.16.1 */ 1215 if(config->ftp_ssl_ccc) 1216 my_setopt_enum(curl, CURLOPT_FTP_SSL_CCC, 1217 (long)config->ftp_ssl_ccc_mode); 1218 1219 /* new in curl 7.19.4 */ 1220 if(config->socks5_gssapi_nec) 1221 my_setopt_str(curl, CURLOPT_SOCKS5_GSSAPI_NEC, 1222 config->socks5_gssapi_nec); 1223 1224 /* new in curl 7.43.0 */ 1225 if(config->proxy_service_name) 1226 my_setopt_str(curl, CURLOPT_PROXY_SERVICE_NAME, 1227 config->proxy_service_name); 1228 1229 /* new in curl 7.43.0 */ 1230 if(config->service_name) 1231 my_setopt_str(curl, CURLOPT_SERVICE_NAME, 1232 config->service_name); 1233 1234 /* curl 7.13.0 */ 1235 my_setopt_str(curl, CURLOPT_FTP_ACCOUNT, config->ftp_account); 1236 1237 my_setopt(curl, CURLOPT_IGNORE_CONTENT_LENGTH, config->ignorecl?1L:0L); 1238 1239 /* curl 7.14.2 */ 1240 my_setopt(curl, CURLOPT_FTP_SKIP_PASV_IP, config->ftp_skip_ip?1L:0L); 1241 1242 /* curl 7.15.1 */ 1243 my_setopt(curl, CURLOPT_FTP_FILEMETHOD, (long)config->ftp_filemethod); 1244 1245 /* curl 7.15.2 */ 1246 if(config->localport) { 1247 my_setopt(curl, CURLOPT_LOCALPORT, (long)config->localport); 1248 my_setopt_str(curl, CURLOPT_LOCALPORTRANGE, 1249 (long)config->localportrange); 1250 } 1251 1252 /* curl 7.15.5 */ 1253 my_setopt_str(curl, CURLOPT_FTP_ALTERNATIVE_TO_USER, 1254 config->ftp_alternative_to_user); 1255 1256 /* curl 7.16.0 */ 1257 if(config->disable_sessionid) 1258 /* disable it */ 1259 my_setopt(curl, CURLOPT_SSL_SESSIONID_CACHE, 0L); 1260 1261 /* curl 7.16.2 */ 1262 if(config->raw) { 1263 my_setopt(curl, CURLOPT_HTTP_CONTENT_DECODING, 0L); 1264 my_setopt(curl, CURLOPT_HTTP_TRANSFER_DECODING, 0L); 1265 } 1266 1267 /* curl 7.17.1 */ 1268 if(!config->nokeepalive) { 1269 my_setopt(curl, CURLOPT_TCP_KEEPALIVE, 1L); 1270 if(config->alivetime != 0) { 1271 my_setopt(curl, CURLOPT_TCP_KEEPIDLE, config->alivetime); 1272 my_setopt(curl, CURLOPT_TCP_KEEPINTVL, config->alivetime); 1273 } 1274 } 1275 else 1276 my_setopt(curl, CURLOPT_TCP_KEEPALIVE, 0L); 1277 1278 /* curl 7.20.0 */ 1279 if(config->tftp_blksize) 1280 my_setopt(curl, CURLOPT_TFTP_BLKSIZE, config->tftp_blksize); 1281 1282 if(config->mail_from) 1283 my_setopt_str(curl, CURLOPT_MAIL_FROM, config->mail_from); 1284 1285 if(config->mail_rcpt) 1286 my_setopt_slist(curl, CURLOPT_MAIL_RCPT, config->mail_rcpt); 1287 1288 /* curl 7.20.x */ 1289 if(config->ftp_pret) 1290 my_setopt(curl, CURLOPT_FTP_USE_PRET, 1L); 1291 1292 if(config->proto_present) 1293 my_setopt_flags(curl, CURLOPT_PROTOCOLS, config->proto); 1294 if(config->proto_redir_present) 1295 my_setopt_flags(curl, CURLOPT_REDIR_PROTOCOLS, config->proto_redir); 1296 1297 if(config->content_disposition 1298 && (urlnode->flags & GETOUT_USEREMOTE)) 1299 hdrcbdata.honor_cd_filename = TRUE; 1300 else 1301 hdrcbdata.honor_cd_filename = FALSE; 1302 1303 hdrcbdata.outs = &outs; 1304 hdrcbdata.heads = &heads; 1305 1306 my_setopt(curl, CURLOPT_HEADERFUNCTION, tool_header_cb); 1307 my_setopt(curl, CURLOPT_HEADERDATA, &hdrcbdata); 1308 1309 if(config->resolve) 1310 /* new in 7.21.3 */ 1311 my_setopt_slist(curl, CURLOPT_RESOLVE, config->resolve); 1312 1313 if(config->connect_to) 1314 /* new in 7.49.0 */ 1315 my_setopt_slist(curl, CURLOPT_CONNECT_TO, config->connect_to); 1316 1317 /* new in 7.21.4 */ 1318 if(curlinfo->features & CURL_VERSION_TLSAUTH_SRP) { 1319 if(config->tls_username) 1320 my_setopt_str(curl, CURLOPT_TLSAUTH_USERNAME, 1321 config->tls_username); 1322 if(config->tls_password) 1323 my_setopt_str(curl, CURLOPT_TLSAUTH_PASSWORD, 1324 config->tls_password); 1325 if(config->tls_authtype) 1326 my_setopt_str(curl, CURLOPT_TLSAUTH_TYPE, 1327 config->tls_authtype); 1328 } 1329 1330 /* new in 7.22.0 */ 1331 if(config->gssapi_delegation) 1332 my_setopt_str(curl, CURLOPT_GSSAPI_DELEGATION, 1333 config->gssapi_delegation); 1334 1335 /* new in 7.25.0 and 7.44.0 */ 1336 { 1337 long mask = (config->ssl_allow_beast ? CURLSSLOPT_ALLOW_BEAST : 0) | 1338 (config->ssl_no_revoke ? CURLSSLOPT_NO_REVOKE : 0); 1339 if(mask) 1340 my_setopt_bitmask(curl, CURLOPT_SSL_OPTIONS, mask); 1341 } 1342 1343 if(config->mail_auth) 1344 my_setopt_str(curl, CURLOPT_MAIL_AUTH, config->mail_auth); 1345 1346 /* new in 7.31.0 */ 1347 if(config->sasl_ir) 1348 my_setopt(curl, CURLOPT_SASL_IR, 1L); 1349 1350 if(config->nonpn) { 1351 my_setopt(curl, CURLOPT_SSL_ENABLE_NPN, 0L); 1352 } 1353 1354 if(config->noalpn) { 1355 my_setopt(curl, CURLOPT_SSL_ENABLE_ALPN, 0L); 1356 } 1357 1358 /* new in 7.40.0 */ 1359 if(config->unix_socket_path) 1360 my_setopt_str(curl, CURLOPT_UNIX_SOCKET_PATH, 1361 config->unix_socket_path); 1362 1363 /* new in 7.45.0 */ 1364 if(config->proto_default) 1365 my_setopt_str(curl, CURLOPT_DEFAULT_PROTOCOL, config->proto_default); 1366 1367 /* new in 7.47.0 */ 1368 if(config->expect100timeout > 0) 1369 my_setopt_str(curl, CURLOPT_EXPECT_100_TIMEOUT_MS, 1370 (long)(config->expect100timeout*1000)); 1371 1372 /* new in 7.48.0 */ 1373 if(config->tftp_no_options) 1374 my_setopt(curl, CURLOPT_TFTP_NO_OPTIONS, 1L); 1375 1376 /* initialize retry vars for loop below */ 1377 retry_sleep_default = (config->retry_delay) ? 1378 config->retry_delay*1000L : RETRY_SLEEP_DEFAULT; /* ms */ 1379 1380 retry_numretries = config->req_retry; 1381 retry_sleep = retry_sleep_default; /* ms */ 1382 retrystart = tvnow(); 1383 1384#ifndef CURL_DISABLE_LIBCURL_OPTION 1385 if(global->libcurl) { 1386 result = easysrc_perform(); 1387 if(result) 1388 goto show_error; 1389 } 1390#endif 1391 1392 for(;;) { 1393#ifdef USE_METALINK 1394 if(!metalink && config->use_metalink) { 1395 /* If outs.metalink_parser is non-NULL, delete it first. */ 1396 if(outs.metalink_parser) 1397 metalink_parser_context_delete(outs.metalink_parser); 1398 outs.metalink_parser = metalink_parser_context_new(); 1399 if(outs.metalink_parser == NULL) { 1400 result = CURLE_OUT_OF_MEMORY; 1401 goto show_error; 1402 } 1403 fprintf(config->global->errors, 1404 "Metalink: parsing (%s) metalink/XML...\n", this_url); 1405 } 1406 else if(metalink) 1407 fprintf(config->global->errors, 1408 "Metalink: fetching (%s) from (%s)...\n", 1409 mlfile->filename, this_url); 1410#endif /* USE_METALINK */ 1411 1412#ifdef CURLDEBUG 1413 if(config->test_event_based) 1414 result = curl_easy_perform_ev(curl); 1415 else 1416#endif 1417 result = curl_easy_perform(curl); 1418 1419 if(!result && !outs.stream && !outs.bytes) { 1420 /* we have received no data despite the transfer was successful 1421 ==> force cration of an empty output file (if an output file 1422 was specified) */ 1423 long cond_unmet = 0L; 1424 /* do not create (or even overwrite) the file in case we get no 1425 data because of unmet condition */ 1426 curl_easy_getinfo(curl, CURLINFO_CONDITION_UNMET, &cond_unmet); 1427 if(!cond_unmet && !tool_create_output_file(&outs)) 1428 result = CURLE_WRITE_ERROR; 1429 } 1430 1431 if(outs.is_cd_filename && outs.stream && !global->mute && 1432 outs.filename) 1433 printf("curl: Saved to filename '%s'\n", outs.filename); 1434 1435 /* if retry-max-time is non-zero, make sure we haven't exceeded the 1436 time */ 1437 if(retry_numretries && 1438 (!config->retry_maxtime || 1439 (tvdiff(tvnow(), retrystart) < 1440 config->retry_maxtime*1000L)) ) { 1441 enum { 1442 RETRY_NO, 1443 RETRY_TIMEOUT, 1444 RETRY_HTTP, 1445 RETRY_FTP, 1446 RETRY_LAST /* not used */ 1447 } retry = RETRY_NO; 1448 long response; 1449 if((CURLE_OPERATION_TIMEDOUT == result) || 1450 (CURLE_COULDNT_RESOLVE_HOST == result) || 1451 (CURLE_COULDNT_RESOLVE_PROXY == result) || 1452 (CURLE_FTP_ACCEPT_TIMEOUT == result)) 1453 /* retry timeout always */ 1454 retry = RETRY_TIMEOUT; 1455 else if((CURLE_OK == result) || 1456 (config->failonerror && 1457 (CURLE_HTTP_RETURNED_ERROR == result))) { 1458 /* If it returned OK. _or_ failonerror was enabled and it 1459 returned due to such an error, check for HTTP transient 1460 errors to retry on. */ 1461 char *effective_url = NULL; 1462 curl_easy_getinfo(curl, CURLINFO_EFFECTIVE_URL, &effective_url); 1463 if(effective_url && 1464 checkprefix("http", effective_url)) { 1465 /* This was HTTP(S) */ 1466 curl_easy_getinfo(curl, CURLINFO_RESPONSE_CODE, &response); 1467 1468 switch(response) { 1469 case 500: /* Internal Server Error */ 1470 case 502: /* Bad Gateway */ 1471 case 503: /* Service Unavailable */ 1472 case 504: /* Gateway Timeout */ 1473 retry = RETRY_HTTP; 1474 /* 1475 * At this point, we have already written data to the output 1476 * file (or terminal). If we write to a file, we must rewind 1477 * or close/re-open the file so that the next attempt starts 1478 * over from the beginning. 1479 * 1480 * TODO: similar action for the upload case. We might need 1481 * to start over reading from a previous point if we have 1482 * uploaded something when this was returned. 1483 */ 1484 break; 1485 } 1486 } 1487 } /* if CURLE_OK */ 1488 else if(result) { 1489 curl_easy_getinfo(curl, CURLINFO_RESPONSE_CODE, &response); 1490 1491 if(response/100 == 4) 1492 /* 1493 * This is typically when the FTP server only allows a certain 1494 * amount of users and we are not one of them. All 4xx codes 1495 * are transient. 1496 */ 1497 retry = RETRY_FTP; 1498 } 1499 1500 if(retry) { 1501 static const char * const m[]={ 1502 NULL, "timeout", "HTTP error", "FTP error" 1503 }; 1504 1505 warnf(config->global, "Transient problem: %s " 1506 "Will retry in %ld seconds. " 1507 "%ld retries left.\n", 1508 m[retry], retry_sleep/1000L, retry_numretries); 1509 1510 tool_go_sleep(retry_sleep); 1511 retry_numretries--; 1512 if(!config->retry_delay) { 1513 retry_sleep *= 2; 1514 if(retry_sleep > RETRY_SLEEP_MAX) 1515 retry_sleep = RETRY_SLEEP_MAX; 1516 } 1517 if(outs.bytes && outs.filename && outs.stream) { 1518 /* We have written data to a output file, we truncate file 1519 */ 1520 if(!global->mute) 1521 fprintf(global->errors, "Throwing away %" 1522 CURL_FORMAT_CURL_OFF_T " bytes\n", 1523 outs.bytes); 1524 fflush(outs.stream); 1525 /* truncate file at the position where we started appending */ 1526#ifdef HAVE_FTRUNCATE 1527 if(ftruncate(fileno(outs.stream), outs.init)) { 1528 /* when truncate fails, we can't just append as then we'll 1529 create something strange, bail out */ 1530 if(!global->mute) 1531 fprintf(global->errors, 1532 "failed to truncate, exiting\n"); 1533 result = CURLE_WRITE_ERROR; 1534 goto quit_urls; 1535 } 1536 /* now seek to the end of the file, the position where we 1537 just truncated the file in a large file-safe way */ 1538 fseek(outs.stream, 0, SEEK_END); 1539#else 1540 /* ftruncate is not available, so just reposition the file 1541 to the location we would have truncated it. This won't 1542 work properly with large files on 32-bit systems, but 1543 most of those will have ftruncate. */ 1544 fseek(outs.stream, (long)outs.init, SEEK_SET); 1545#endif 1546 outs.bytes = 0; /* clear for next round */ 1547 } 1548 continue; /* curl_easy_perform loop */ 1549 } 1550 } /* if retry_numretries */ 1551 else if(metalink) { 1552 /* Metalink: Decide to try the next resource or 1553 not. Basically, we want to try the next resource if 1554 download was not successful. */ 1555 long response; 1556 if(CURLE_OK == result) { 1557 /* TODO We want to try next resource when download was 1558 not successful. How to know that? */ 1559 char *effective_url = NULL; 1560 curl_easy_getinfo(curl, CURLINFO_EFFECTIVE_URL, &effective_url); 1561 if(effective_url && 1562 curlx_strnequal(effective_url, "http", 4)) { 1563 /* This was HTTP(S) */ 1564 curl_easy_getinfo(curl, CURLINFO_RESPONSE_CODE, &response); 1565 if(response != 200 && response != 206) { 1566 metalink_next_res = 1; 1567 fprintf(global->errors, 1568 "Metalink: fetching (%s) from (%s) FAILED " 1569 "(HTTP status code %d)\n", 1570 mlfile->filename, this_url, response); 1571 } 1572 } 1573 } 1574 else { 1575 metalink_next_res = 1; 1576 fprintf(global->errors, 1577 "Metalink: fetching (%s) from (%s) FAILED (%s)\n", 1578 mlfile->filename, this_url, 1579 (errorbuffer[0]) ? 1580 errorbuffer : curl_easy_strerror(result)); 1581 } 1582 } 1583 if(metalink && !metalink_next_res) 1584 fprintf(global->errors, "Metalink: fetching (%s) from (%s) OK\n", 1585 mlfile->filename, this_url); 1586 1587 /* In all ordinary cases, just break out of loop here */ 1588 break; /* curl_easy_perform loop */ 1589 1590 } 1591 1592 if((global->progressmode == CURL_PROGRESS_BAR) && 1593 progressbar.calls) 1594 /* if the custom progress bar has been displayed, we output a 1595 newline here */ 1596 fputs("\n", progressbar.out); 1597 1598 if(config->writeout) 1599 ourWriteOut(curl, &outs, config->writeout); 1600 1601 if(config->writeenv) 1602 ourWriteEnv(curl); 1603 1604 /* 1605 ** Code within this loop may jump directly here to label 'show_error' 1606 ** in order to display an error message for CURLcode stored in 'res' 1607 ** variable and exit loop once that necessary writing and cleanup 1608 ** in label 'quit_urls' has been done. 1609 */ 1610 1611 show_error: 1612 1613#ifdef __VMS 1614 if(is_vms_shell()) { 1615 /* VMS DCL shell behavior */ 1616 if(!global->showerror) 1617 vms_show = VMSSTS_HIDE; 1618 } 1619 else 1620#endif 1621 if(result && global->showerror) { 1622 fprintf(global->errors, "curl: (%d) %s\n", result, (errorbuffer[0]) ? 1623 errorbuffer : curl_easy_strerror(result)); 1624 if(result == CURLE_SSL_CACERT) 1625 fprintf(global->errors, "%s%s", 1626 CURL_CA_CERT_ERRORMSG1, CURL_CA_CERT_ERRORMSG2); 1627 } 1628 1629 /* Fall through comment to 'quit_urls' label */ 1630 1631 /* 1632 ** Upon error condition and always that a message has already been 1633 ** displayed, code within this loop may jump directly here to label 1634 ** 'quit_urls' otherwise it should jump to 'show_error' label above. 1635 ** 1636 ** When 'res' variable is _not_ CURLE_OK loop will exit once that 1637 ** all code following 'quit_urls' has been executed. Otherwise it 1638 ** will loop to the beginning from where it may exit if there are 1639 ** no more urls left. 1640 */ 1641 1642 quit_urls: 1643 1644 /* Set file extended attributes */ 1645 if(!result && config->xattr && outs.fopened && outs.stream) { 1646 int rc = fwrite_xattr(curl, fileno(outs.stream)); 1647 if(rc) 1648 warnf(config->global, "Error setting extended attributes: %s\n", 1649 strerror(errno)); 1650 } 1651 1652 /* Close the file */ 1653 if(outs.fopened && outs.stream) { 1654 int rc = fclose(outs.stream); 1655 if(!result && rc) { 1656 /* something went wrong in the writing process */ 1657 result = CURLE_WRITE_ERROR; 1658 fprintf(global->errors, "(%d) Failed writing body\n", result); 1659 } 1660 } 1661 else if(!outs.s_isreg && outs.stream) { 1662 /* Dump standard stream buffered data */ 1663 int rc = fflush(outs.stream); 1664 if(!result && rc) { 1665 /* something went wrong in the writing process */ 1666 result = CURLE_WRITE_ERROR; 1667 fprintf(global->errors, "(%d) Failed writing body\n", result); 1668 } 1669 } 1670 1671#ifdef __AMIGA__ 1672 if(!result && outs.s_isreg && outs.filename) { 1673 /* Set the url (up to 80 chars) as comment for the file */ 1674 if(strlen(url) > 78) 1675 url[79] = '\0'; 1676 SetComment(outs.filename, url); 1677 } 1678#endif 1679 1680#ifdef HAVE_UTIME 1681 /* File time can only be set _after_ the file has been closed */ 1682 if(!result && config->remote_time && outs.s_isreg && outs.filename) { 1683 /* Ask libcurl if we got a remote file time */ 1684 long filetime = -1; 1685 curl_easy_getinfo(curl, CURLINFO_FILETIME, &filetime); 1686 if(filetime >= 0) { 1687 struct utimbuf times; 1688 times.actime = (time_t)filetime; 1689 times.modtime = (time_t)filetime; 1690 utime(outs.filename, ×); /* set the time we got */ 1691 } 1692 } 1693#endif 1694 1695#ifdef USE_METALINK 1696 if(!metalink && config->use_metalink && result == CURLE_OK) { 1697 int rv = parse_metalink(config, &outs, this_url); 1698 if(rv == 0) 1699 fprintf(config->global->errors, "Metalink: parsing (%s) OK\n", 1700 this_url); 1701 else if(rv == -1) 1702 fprintf(config->global->errors, "Metalink: parsing (%s) FAILED\n", 1703 this_url); 1704 } 1705 else if(metalink && result == CURLE_OK && !metalink_next_res) { 1706 int rv = metalink_check_hash(global, mlfile, outs.filename); 1707 if(rv == 0) { 1708 metalink_next_res = 1; 1709 } 1710 } 1711#endif /* USE_METALINK */ 1712 1713 /* No more business with this output struct */ 1714 if(outs.alloc_filename) 1715 Curl_safefree(outs.filename); 1716#ifdef USE_METALINK 1717 if(outs.metalink_parser) 1718 metalink_parser_context_delete(outs.metalink_parser); 1719#endif /* USE_METALINK */ 1720 memset(&outs, 0, sizeof(struct OutStruct)); 1721 hdrcbdata.outs = NULL; 1722 1723 /* Free loop-local allocated memory and close loop-local opened fd */ 1724 1725 Curl_safefree(outfile); 1726 Curl_safefree(this_url); 1727 1728 if(infdopen) 1729 close(infd); 1730 1731 if(metalink) { 1732 /* Should exit if error is fatal. */ 1733 if(is_fatal_error(result)) { 1734 break; 1735 } 1736 if(!metalink_next_res) 1737 break; 1738 mlres = mlres->next; 1739 if(mlres == NULL) 1740 /* TODO If metalink_next_res is 1 and mlres is NULL, 1741 * set res to error code 1742 */ 1743 break; 1744 } 1745 else 1746 if(urlnum > 1) { 1747 /* when url globbing, exit loop upon critical error */ 1748 if(is_fatal_error(result)) 1749 break; 1750 } 1751 else if(result) 1752 /* when not url globbing, exit loop upon any error */ 1753 break; 1754 1755 } /* loop to the next URL */ 1756 1757 /* Free loop-local allocated memory */ 1758 1759 Curl_safefree(uploadfile); 1760 1761 if(urls) { 1762 /* Free list of remaining URLs */ 1763 glob_cleanup(urls); 1764 urls = NULL; 1765 } 1766 1767 if(infilenum > 1) { 1768 /* when file globbing, exit loop upon critical error */ 1769 if(is_fatal_error(result)) 1770 break; 1771 } 1772 else if(result) 1773 /* when not file globbing, exit loop upon any error */ 1774 break; 1775 1776 } /* loop to the next globbed upload file */ 1777 1778 /* Free loop-local allocated memory */ 1779 1780 Curl_safefree(outfiles); 1781 1782 if(inglob) { 1783 /* Free list of globbed upload files */ 1784 glob_cleanup(inglob); 1785 inglob = NULL; 1786 } 1787 1788 /* Free this URL node data without destroying the 1789 the node itself nor modifying next pointer. */ 1790 Curl_safefree(urlnode->url); 1791 Curl_safefree(urlnode->outfile); 1792 Curl_safefree(urlnode->infile); 1793 urlnode->flags = 0; 1794 1795 /* 1796 ** Bail out upon critical errors 1797 */ 1798 if(is_fatal_error(result)) 1799 goto quit_curl; 1800 1801 } /* for-loop through all URLs */ 1802 1803 /* 1804 ** Nested loops end here. 1805 */ 1806 1807 quit_curl: 1808 1809 /* Reset the global config variables */ 1810 global->noprogress = orig_noprogress; 1811 global->isatty = orig_isatty; 1812 1813 /* Free function-local referenced allocated memory */ 1814 Curl_safefree(httpgetfields); 1815 1816 /* Free list of given URLs */ 1817 clean_getout(config); 1818 1819 hdrcbdata.heads = NULL; 1820 1821 /* Close function-local opened file descriptors */ 1822 if(heads.fopened && heads.stream) 1823 fclose(heads.stream); 1824 1825 if(heads.alloc_filename) 1826 Curl_safefree(heads.filename); 1827 1828 /* Release metalink related resources here */ 1829 clean_metalink(config); 1830 1831 return result; 1832} 1833 1834CURLcode operate(struct GlobalConfig *config, int argc, argv_item_t argv[]) 1835{ 1836 CURLcode result = CURLE_OK; 1837 1838 /* Setup proper locale from environment */ 1839#ifdef HAVE_SETLOCALE 1840 setlocale(LC_ALL, ""); 1841#endif 1842 1843 /* Parse .curlrc if necessary */ 1844 if((argc == 1) || 1845 (!curlx_strequal(argv[1], "-q") && 1846 !curlx_strequal(argv[1], "--disable"))) { 1847 parseconfig(NULL, config); /* ignore possible failure */ 1848 1849 /* If we had no arguments then make sure a url was specified in .curlrc */ 1850 if((argc < 2) && (!config->first->url_list)) { 1851 helpf(config->errors, NULL); 1852 result = CURLE_FAILED_INIT; 1853 } 1854 } 1855 1856 if(!result) { 1857 /* Parse the command line arguments */ 1858 ParameterError res = parse_args(config, argc, argv); 1859 if(res) { 1860 result = CURLE_OK; 1861 1862 /* Check if we were asked for the help */ 1863 if(res == PARAM_HELP_REQUESTED) 1864 tool_help(); 1865 /* Check if we were asked for the manual */ 1866 else if(res == PARAM_MANUAL_REQUESTED) 1867 hugehelp(); 1868 /* Check if we were asked for the version information */ 1869 else if(res == PARAM_VERSION_INFO_REQUESTED) 1870 tool_version_info(); 1871 /* Check if we were asked to list the SSL engines */ 1872 else if(res == PARAM_ENGINES_REQUESTED) 1873 tool_list_engines(config->easy); 1874 else if(res == PARAM_LIBCURL_UNSUPPORTED_PROTOCOL) 1875 result = CURLE_UNSUPPORTED_PROTOCOL; 1876 else 1877 result = CURLE_FAILED_INIT; 1878 } 1879 else { 1880#ifndef CURL_DISABLE_LIBCURL_OPTION 1881 if(config->libcurl) { 1882 /* Initialise the libcurl source output */ 1883 result = easysrc_init(); 1884 } 1885#endif 1886 1887 /* Perform the main operations */ 1888 if(!result) { 1889 size_t count = 0; 1890 struct OperationConfig *operation = config->first; 1891 1892 /* Get the required aguments for each operation */ 1893 while(!result && operation) { 1894 result = get_args(operation, count++); 1895 1896 operation = operation->next; 1897 } 1898 1899 /* Set the current operation pointer */ 1900 config->current = config->first; 1901 1902 /* Perform each operation */ 1903 while(!result && config->current) { 1904 result = operate_do(config, config->current); 1905 1906 config->current = config->current->next; 1907 } 1908 1909#ifndef CURL_DISABLE_LIBCURL_OPTION 1910 if(config->libcurl) { 1911 /* Cleanup the libcurl source output */ 1912 easysrc_cleanup(); 1913 1914 /* Dump the libcurl code if previously enabled */ 1915 dumpeasysrc(config); 1916 } 1917#endif 1918 } 1919 else 1920 helpf(config->errors, "out of memory\n"); 1921 } 1922 } 1923 1924 return result; 1925} 1926