18d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi# -*- coding: utf-8 -*- 28d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi# Copyright 2013 Google Inc. All Rights Reserved. 38d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi# 48d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi# Licensed under the Apache License, Version 2.0 (the "License"); 58d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi# you may not use this file except in compliance with the License. 68d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi# You may obtain a copy of the License at 78d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi# 88d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi# http://www.apache.org/licenses/LICENSE-2.0 98d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi# 108d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi# Unless required by applicable law or agreed to in writing, software 118d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi# distributed under the License is distributed on an "AS IS" BASIS, 128d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 138d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi# See the License for the specific language governing permissions and 148d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi# limitations under the License. 158d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi"""Additional help about types of credentials and authentication.""" 168d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi 178d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoifrom __future__ import absolute_import 188d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi 198d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoifrom gslib.help_provider import HelpProvider 208d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi 218d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi_DETAILED_HELP_TEXT = (""" 228d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi<B>OVERVIEW</B> 238d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi gsutil currently supports several types of credentials/authentication, as 248d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi well as the ability to access public data anonymously (see "gsutil help anon" 258d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi for more on anonymous access). Each of these type of credentials is discussed 268d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi in more detail below, along with information about configuring and using 278d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi credentials via either the Cloud SDK or standalone installations of gsutil. 288d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi 298d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi 308d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi<B>Configuring/Using Credentials via Cloud SDK Distribution of gsutil</B> 318d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi When gsutil is installed/used via the Cloud SDK ("gcloud"), credentials are 328d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi stored by Cloud SDK in a non-user-editable file located under 338d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi ~/.config/gcloud (any manipulation of credentials should be done via the 348d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi gcloud auth command). If you need to set up multiple credentials (e.g., one 358d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi for an individual user account and a second for a service account), the 368d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi gcloud auth command manages the credentials for you, and you switch between 378d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi credentials using the gcloud auth command as well (for more details see 388d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi https://developers.google.com/cloud/sdk/gcloud/#gcloud.auth). 398d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi 408d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi Once credentials have been configured via gcloud auth, those credentials will 418d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi be used regardless of whether the user has any boto configuration files (which 428d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi are located at ~/.boto unless a different path is specified in the BOTO_CONFIG 438d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi environment variable). However, gsutil will still look for credentials in the 448d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi boto config file if a type of credential is needed that's not stored in the 458d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi gcloud credential store (e.g., an HMAC credential for an S3 account). 468d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi 478d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi 488d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi<B>Configuring/Using Credentials via Standalone gsutil Distribution</B> 498d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi If you installed a standalone distribution of gsutil (downloaded from 508d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi https://pub.storage.googleapis.com/gsutil.tar.gz, 518d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi https://pub.storage.googleapis.com/gsutil.zip, or PyPi), credentials are 528d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi configured using the gsutil config command, and are stored in the 538d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi user-editable boto config file (located at ~/.boto unless a different path is 548d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi specified in the BOTO_CONFIG environment). In this case if you want to set up 558d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi multiple credentials (e.g., one for an individual user account and a second 568d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi for a service account), you run gsutil config once for each credential, and 578d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi save each of the generated boto config files (e.g., renaming one to 588d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi ~/.boto_user_account and the second to ~/.boto_service_account), and you 598d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi switch between the credentials using the BOTO_CONFIG environment variable 608d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi (e.g., by running BOTO_CONFIG=~/.boto_user_account gsutil ls). 618d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi 628d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi Note that when using the standalone version of gsutil with the JSON API you 638d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi can configure at most one of the following types of GCS credentials in a 648d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi single boto config file: OAuth2 User Account, OAuth2 Service Account. In 658d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi addition to these, you may also have S3 HMAC credentials (necessary for using 668d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi s3:// URLs) and GCE Internal Service Account credentials. GCE Internal Service 678d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi Account credentials are used only when OAuth2 credentials are not present. 688d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi 698d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi 708d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi<B>SUPPORTED CREDENTIAL TYPES</B> 718d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi gsutil supports several types of credentials (the specific subset depends on 728d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi which distribution of gsutil you are using; see above discussion). 738d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi 748d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi OAuth2 User Account: 758d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi This is the preferred type of credentials for authenticating requests on 768d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi behalf of a specific user (which is probably the most common use of gsutil). 778d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi This is the default type of credential that will be created when you run 788d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi "gsutil config". 798d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi For more details about OAuth2 authentication, see: 808d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi https://developers.google.com/accounts/docs/OAuth2#scenarios 818d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi 828d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi HMAC: 838d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi This type of credential can be used by programs that are implemented using 848d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi HMAC authentication, which is an authentication mechanism supported by 858d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi certain other cloud storage service providers. This type of credential can 868d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi also be used for interactive use when moving data to/from service providers 878d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi that support HMAC credentials. This is the type of credential that will be 888d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi created when you run "gsutil config -a". 898d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi 908d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi Note that it's possible to set up HMAC credentials for both Google Cloud 918d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi Storage and another service provider; or to set up OAuth2 user account 928d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi credentials for Google Cloud Storage and HMAC credentials for another 938d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi service provider. To do so, after you run the gsutil config command, you 948d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi can edit the generated ~/.boto config file and look for comments for where 958d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi other credentials can be added. 968d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi 978d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi For more details about HMAC authentication, see: 988d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi https://developers.google.com/storage/docs/reference/v1/getting-startedv1#keys 998d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi 1008d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi OAuth2 Service Account: 1018d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi This is the preferred type of credential to use when authenticating on 1028d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi behalf of a service or application (as opposed to a user). For example, if 1038d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi you will run gsutil out of a nightly cron job to upload/download data, 1048d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi using a service account allows the cron job not to depend on credentials of 1058d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi an individual employee at your company. This is the type of credential that 1068d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi will be configured when you run "gsutil config -e". 1078d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi 1088d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi It is important to note that a service account is considered an Editor by 1098d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi default for the purposes of API access, rather than an Owner. In particular, 1108d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi the fact that Editors have OWNER access in the default object and 1118d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi bucket ACLs, but the canned ACL options remove OWNER access from 1128d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi Editors, can lead to unexpected results. The solution to this problem is to 1138d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi add the email address for your service account as a project editor. To find 1148d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi the email address, visit the 1158d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi `Google Developers Console <https://cloud.google.com/console#/project>`_, 1168d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi click on the project you're using, click "APIs & auth", and click 1178d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi "Credentials". 1188d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi 1198d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi To create a service account, visit the Google Developers Console and then: 1208d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi 1218d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi - Click "APIs & auth" in the left sidebar. 1228d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi 1238d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi - Click "Credentials". 1248d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi 1258d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi - Click "Create New Client ID". 1268d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi 1278d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi - Select "Service Account" as your application type. 1288d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi 1298d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi - Save the JSON private key or the .p12 private key and password 1308d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi provided. 1318d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi 1328d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi For further information about account roles, see: 1338d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi https://developers.google.com/console/help/#DifferentRoles 1348d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi 1358d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi For more details about OAuth2 service accounts, see: 1368d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi https://developers.google.com/accounts/docs/OAuth2ServiceAccount 1378d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi 1388d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi GCE Internal Service Account: 1398d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi This is the type of service account used for accounts hosted by App Engine 1408d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi or GCE. Such credentials are created automatically for you on GCE when you 1418d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi run the gcutil addinstance command with the --service_account flag. 1428d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi 1438d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi For more details about GCE service accounts, see: 1448d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi https://developers.google.com/compute/docs/authentication; 1458d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi 1468d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi For more details about App Engine service accounts, see: 1478d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi https://developers.google.com/appengine/docs/python/appidentity/overview 1488d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi""") 1498d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi 1508d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi 1518d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoiclass CommandOptions(HelpProvider): 1528d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi """Additional help about types of credentials and authentication.""" 1538d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi 1548d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi # Help specification. See help_provider.py for documentation. 1558d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi help_spec = HelpProvider.HelpSpec( 1568d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi help_name='creds', 1578d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi help_name_aliases=['credentials', 'authentication', 'auth', 'gcloud'], 1588d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi help_type='additional_help', 1598d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi help_one_line_summary='Credential Types Supporting Various Use Cases', 1608d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi help_text=_DETAILED_HELP_TEXT, 1618d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi subcommand_help_text={}, 1628d2b206a675ec20ea07100c35df34e65ee1e45e8Ruchi Kandoi ) 163