davidkotnik/novafarma
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: Add new gcloud commands, API clients, and third-party libraries across various services.

Browse Source
This commit is contained in:
David Kotnik
2026-01-01 20:26:35 +01:00
parent 5e23cbece0
commit a19e592eb7
25221 changed files with 8324611 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

20
gcloud auth application-default login/google-cloud-sdk/platform/gsutil/gslib/addlhelp/__init__.py Normal file
View File

@@ -0,0 +1,20 @@
# -*- coding: utf-8 -*-
# Copyright 2012 Google Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Package marker file."""
from __future__ import absolute_import
from __future__ import print_function
from __future__ import division
from __future__ import unicode_literals

240
gcloud auth application-default login/google-cloud-sdk/platform/gsutil/gslib/addlhelp/acls.py Normal file
View File

@@ -0,0 +1,240 @@
# -*- coding: utf-8 -*-
# Copyright 2012 Google Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Additional help about Access Control Lists."""
# TODO(iam-beta): Revise this to include new IAM syntax.
from __future__ import absolute_import
from __future__ import print_function
from __future__ import division
from __future__ import unicode_literals
from gslib.help_provider import HelpProvider
_DETAILED_HELP_TEXT = ("""
<B>OVERVIEW</B>
Access Control Lists (ACLs) allow you to control who can read and write
your data, and who can read and write the ACLs themselves.
If not specified at the time an object is uploaded (e.g., via the gsutil cp
-a option), objects will be created with a default object ACL set on the
bucket (see "gsutil help defacl"). You can replace the ACL on an object
or bucket using the "gsutil acl set" command, or
modify the existing ACL using the "gsutil acl ch" command (see "gsutil help
acl").
<B>BUCKET VS OBJECT ACLS</B>
In Google Cloud Storage, the bucket ACL works as follows:
- Users granted READ access are allowed to list the bucket contents and read
bucket metadata other than its ACL.
- Users granted WRITE access are allowed READ access and also are allowed to
write and delete objects in that bucket, including overwriting previously
written objects.
- Users granted OWNER access are allowed WRITE access and also are allowed to
read and write the bucket's ACL.
The object ACL works as follows:
- Users granted READ access are allowed to read the object's data and
metadata.
- Users granted OWNER access are allowed READ access and also are allowed to
read and write the object's ACL.
A couple of points are worth noting, that sometimes surprise users:
1. There is no WRITE access for objects; attempting to set an ACL with WRITE
permission for an object will result in an error.
2. The bucket ACL plays no role in determining who can read objects; only the
object ACL matters for that purpose. This is different from how things
work in Linux file systems, where both the file and directory permission
control file read access. It also means, for example, that someone with
OWNER over the bucket may not have read access to objects in the bucket.
This is by design, and supports useful cases. For example, you might want
to set up bucket ownership so that a small group of administrators have
OWNER on the bucket (with the ability to delete data to control storage
costs), but not grant those users read access to the object data (which
might be sensitive data that should only be accessed by a different
specific group of users).
<B>ACCESSING PUBLIC OBJECTS</B>
Objects with public READ access can be accessed anonymously by gsutil, via
a browser, or via Cloud Storage APIs. For more details on accessing public
objects, see:
https://cloud.google.com/storage/docs/access-public-data
<B>CANNED ACLS</B>
The simplest way to set an ACL on a bucket or object is using a "canned
ACL". The available canned ACLs are:
project-private
Gives permission to the project team based on their roles. Anyone who is
part of the team has READ permission, and project owners and project editors
have OWNER permission. This is the default ACL for newly created
buckets. This is also the default ACL for newly created objects unless the
default object ACL for that bucket has been changed. For more details see
"gsutil help projects".
private
Gives the requester (and only the requester) OWNER permission for a
bucket or object.
public-read
Gives all users (whether logged in or anonymous) READ permission. When
you apply this to an object, anyone on the Internet can read the object
without authenticating.
NOTE: By default, publicly readable objects are served with a Cache-Control
header allowing such objects to be cached for 3600 seconds. If you need to
ensure that updates become visible immediately, you should set a
Cache-Control header of "Cache-Control:private, max-age=0, no-transform" on
such objects. For help doing this, see 'gsutil help setmeta'.
NOTE: Setting a bucket ACL to public-read will remove all OWNER and WRITE
permissions from everyone except the project owner group. Setting an object
ACL to public-read will remove all OWNER and WRITE permissions from
everyone except the object owner. For this reason, we recommend using
the "acl ch" command to make these changes; see "gsutil help acl ch" for
details.
public-read-write
Gives all users READ and WRITE permission. This ACL applies only to buckets.
NOTE: Setting a bucket to public-read-write will allow anyone on the
Internet to upload anything to your bucket. You will be responsible for this
content.
NOTE: Setting a bucket ACL to public-read-write will remove all OWNER
permissions from everyone except the project owner group. Setting an object
ACL to public-read-write will remove all OWNER permissions from
everyone except the object owner. For this reason, we recommend using
the "acl ch" command to make these changes; see "gsutil help acl ch" for
details.
authenticated-read
Gives the requester OWNER permission and gives all authenticated
Google account holders READ permission.
bucket-owner-read
Gives the requester OWNER permission and gives the bucket owner READ
permission. This is used only with objects.
bucket-owner-full-control
Gives the requester OWNER permission and gives the bucket owner
OWNER permission. This is used only with objects.
<B>ACL JSON</B>
When you use a canned ACL, it is translated into an JSON representation
that can later be retrieved and edited to specify more fine-grained
detail about who can read and write buckets and objects. By running
the "gsutil acl get" command you can retrieve the ACL JSON, and edit it to
customize the permissions.
As an example, if you create an object in a bucket that has no default
object ACL set and then retrieve the ACL on the object, it will look
something like this:
[
{
"entity": "group-00b4903a9740e42c29800f53bd5a9a62a2f96eb3f64a4313a115df3f3a776bf7",
"entityId": "00b4903a9740e42c29800f53bd5a9a62a2f96eb3f64a4313a115df3f3a776bf7",
"role": "OWNER"
},
{
"entity": "group-00b4903a977fd817e9da167bc81306489181a110456bb635f466d71cf90a0d51",
"entityId": "00b4903a977fd817e9da167bc81306489181a110456bb635f466d71cf90a0d51",
"role": "OWNER"
},
{
"entity": "00b4903a974898cc8fc309f2f2835308ba3d3df1b889d3fc7e33e187d52d8e71",
"entityId": "00b4903a974898cc8fc309f2f2835308ba3d3df1b889d3fc7e33e187d52d8e71",
"role": "READER"
}
]
The ACL consists collection of elements, each of which specifies an Entity
and a Role. Entities are the way you specify an individual or group of
individuals, and Roles specify what access they're permitted.
This particular ACL grants OWNER to two groups (which means members
of those groups are allowed to read the object and read and write the ACL),
and READ permission to a third group. The project groups are (in order)
the project owners group, editors group, and viewers group.
The 64 digit hex identifiers (following any prefixes like "group-") used in
this ACL are called canonical IDs. They are used to identify predefined
groups associated with the project that owns the bucket: the Project Owners,
Project Editors, and All Project Team Members groups. For more information
the permissions and roles of these project groups, see "gsutil help projects".
Here's an example of an ACL specified using the group-by-email and
group-by-domain entities:
[
{
"entity": "group-travel-companion-owners@googlegroups.com"
"email": "travel-companion-owners@googlegroups.com",
"role": "OWNER",
}
{
"domain": "example.com",
"entity": "domain-example.com"
"role": "READER",
},
]
This ACL grants members of an email group OWNER, and grants READ
access to any user in a domain (which must be a Google Apps for Business
domain). By applying email group grants to a collection of objects
you can edit access control for large numbers of objects at once via
http://groups.google.com. That way, for example, you can easily and quickly
change access to a group of company objects when employees join and leave
your company (i.e., without having to individually change ACLs across
potentially millions of objects).
<B>SHARING SCENARIOS</B>
For more detailed examples how to achieve various useful sharing use
cases see https://cloud.google.com/storage/docs/collaboration
""")
class CommandOptions(HelpProvider):
"""Additional help about Access Control Lists."""
# Help specification. See help_provider.py for documentation.
help_spec = HelpProvider.HelpSpec(
help_name='acls',
help_name_aliases=[
'ACL',
'access control',
'access control list',
'authorization',
'canned',
'canned acl',
],
help_type='additional_help',
help_one_line_summary='Working With Access Control Lists',
help_text=_DETAILED_HELP_TEXT,
subcommand_help_text={},
)

172
gcloud auth application-default login/google-cloud-sdk/platform/gsutil/gslib/addlhelp/command_opts.py Normal file
View File

@@ -0,0 +1,172 @@
# -*- coding: utf-8 -*-
# Copyright 2012 Google Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Additional help about gsutil command-level options."""
from __future__ import absolute_import
from __future__ import print_function
from __future__ import division
from __future__ import unicode_literals
from gslib.help_provider import HelpProvider
_DETAILED_HELP_TEXT = ("""
<B>DESCRIPTION</B>
gsutil supports separate options for the top-level gsutil command and
the individual sub-commands (like cp, rm, etc.) The top-level options
control behavior of gsutil that apply across commands. For example, in
the command:
gsutil -m cp -p file gs://bucket/obj
the -m option applies to gsutil, while the -p option applies to the cp
sub-command.
<B>OPTIONS</B>
-D Shows HTTP requests/headers and additional debug info needed
when posting support requests, including exception stack traces.
CAUTION: The output from using this flag includes authentication
credentials. Before including this flag in your command, be sure
you understand how the command's output is used, and, if
necessary, remove or redact sensitive information.
-DD Same as -D, plus HTTP upstream payload.
-h Allows you to specify certain HTTP headers, for example:
gsutil -h "Cache-Control:public,max-age=3600" \\
-h "Content-Type:text/html" cp ...
Note that you need to quote the headers/values that
contain spaces (such as "Content-Disposition: attachment;
filename=filename.ext"), to avoid having the shell split them
into separate arguments.
The following headers are stored as object metadata and used
in future requests on the object:
Cache-Control
Content-Disposition
Content-Encoding
Content-Language
Content-Type
The following headers are used to check data integrity:
Content-MD5
gsutil also supports custom metadata headers with a matching
Cloud Storage Provider prefix, such as:
x-goog-meta-
Note that for gs:// URLs, the Cache Control header is specific to
the API being used. The XML API accepts any cache control headers
and returns them during object downloads. The JSON API respects
only the public, private, no-cache, max-age, and no-transform
cache control headers.
See "gsutil help setmeta" for the ability to set metadata
fields on objects after they have been uploaded.
-i Allows you to use the configured credentials to impersonate a
service account, for example:
gsutil -i "service-account@google.com" ls gs://pub
Note that this setting will be ignored by the XML API and S3. See
'gsutil help creds' for more information on impersonating service
accounts.
-m Causes supported operations (acl ch, acl set, cp, mv, rm, rsync,
and setmeta) to run in parallel. This can significantly improve
performance if you are performing operations on a large number of
files over a reasonably fast network connection.
gsutil performs the specified operation using a combination of
multi-threading and multi-processing. The number of threads
and processors are determined by ``parallel_thread_count`` and
``parallel_process_count``, respectively. These values are set in
the .boto configuration file or specified in individual requests
with the ``-o`` top-level flag. Because gsutil has no built-in
support for throttling requests, you should experiment with these
values. The optimal values can vary based on a number of factors,
including network speed, number of CPUs, and available memory.
Using the -m option can consume a significant amount of network
bandwidth and cause problems or make your performance worse if
you use a slower network. For example, if you start a large rsync
operation over a network link that's also used by a number of
other important jobs, there could be degraded performance in
those jobs. Similarly, the -m option can make your performance
worse, especially for cases that perform all operations locally,
because it can "thrash" your local disk.
To prevent such issues, reduce the values for
``parallel_thread_count`` and ``parallel_process_count``, or stop
using the -m option entirely. One tool that you can use to limit
how much I/O capacity gsutil consumes and prevent it from
monopolizing your local disk is `ionice
<http://www.tutorialspoint.com/unix_commands/ionice.htm>`_
(built in to many Linux systems). For example, the following
command reduces the I/O priority of gsutil so it doesn't
monopolize your local disk:
ionice -c 2 -n 7 gsutil -m rsync -r ./dir gs://some bucket
If a download or upload operation using parallel transfer fails
before the entire transfer is complete (e.g. failing after 300 of
1000 files have been transferred), you must restart the entire
transfer.
Also, although most commands normally fail upon encountering an
error when the -m flag is disabled, all commands continue to try
all operations when -m is enabled with multiple threads or
processes, and the number of failed operations (if any) are
reported as an exception at the end of the command's execution.
-o Override values in the `boto configuration file
<https://cloud.google.com/storage/docs/boto-gsutil>`_ for the
current command, in the format ``<section>:<name>=<value>``. For
example, ``gsutil -o "GSUtil:parallel_thread_count=4" ...``. This
does not pass the option to gsutil integration tests and does not
change the values that are saved in the boto configuration file.
-q Causes gsutil to perform operations quietly, i.e., without
reporting progress indicators of files being copied or removed,
etc. Errors are still reported. This option can be useful for
running gsutil from a cron job that logs its output to a file, for
which the only information desired in the log is failures.
-u Allows you to specify the ID or number of a user project to be
billed for the request. For example:
gsutil -u "bill-this-project" cp ...
""")
class CommandOptions(HelpProvider):
"""Additional help about gsutil command-level options."""
# Help specification. See help_provider.py for documentation.
help_spec = HelpProvider.HelpSpec(
help_name='options',
help_name_aliases=['arg', 'args', 'cli', 'opt', 'opts'],
help_type='additional_help',
help_one_line_summary='Global Command Line Options',
help_text=_DETAILED_HELP_TEXT,
subcommand_help_text={},
)

169
gcloud auth application-default login/google-cloud-sdk/platform/gsutil/gslib/addlhelp/crc32c.py Normal file
View File

@@ -0,0 +1,169 @@
# -*- coding: utf-8 -*-
# Copyright 2013 Google Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Additional help about CRC32C and installing crcmod."""
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
from __future__ import unicode_literals
from gslib.help_provider import HelpProvider
_DETAILED_HELP_TEXT = ("""
<B>OVERVIEW</B>
Google Cloud Storage provides a cyclic redundancy check (CRC) header that
allows clients to verify the integrity of object contents. For non-composite
objects Google Cloud Storage also provides an MD5 header to allow clients to
verify object integrity, but for composite objects only the CRC is available.
gsutil automatically performs integrity checks on all uploads and downloads.
Additionally, you can use the ``gsutil hash`` command to calculate a CRC for
any local file.
The CRC variant used by Google Cloud Storage is called CRC32C (Castagnoli),
which is not available in the standard Python distribution. The implementation
of CRC32C used by gsutil is provided by a third-party Python module called
`crcmod <https://pypi.python.org/pypi/crcmod>`_.
The crcmod module contains a pure-Python implementation of CRC32C, but using
it results in slow checksum computation and subsequently very poor
performance. A Python C extension is also provided by crcmod, which requires
compiling into a binary module for use. gsutil ships with a precompiled
crcmod C extension for macOS; for other platforms, see the installation
instructions below.
At the end of each copy operation, the ``gsutil cp``, ``gsutil mv``, and
``gsutil rsync`` commands validate that the checksum of the source
file/object matches the checksum of the destination file/object. If the
checksums do not match, gsutil will delete the invalid copy and print a
warning message. This very rarely happens, but if it does, you should
retry the operation.
<B>CONFIGURATION</B>
To determine if the compiled version of crcmod is available in your Python
environment, you can inspect the output of the ``gsutil version`` command for
the "compiled crcmod" entry:
$ gsutil version -l
...
compiled crcmod: True
...
If your crcmod library is compiled to a native binary, this value will be
True. If using the pure-Python version, the value will be False.
To control gsutil's behavior in response to crcmod's status, you can set the
``check_hashes`` variable in your `boto configuration file
<https://cloud.google.com/storage/docs/boto-gsutil>`_. For details on this
variable, see the surrounding comments in your boto configuration file. If
``check_hashes`` is not present in your configuration file, regenerate the
file by running ``gsutil config`` with the appropriate ``-e`` or ``-a`` flag.
<B>INSTALLATION</B>
These installation instructions assume that:
- You have ``pip`` installed. Consult the `pip installation instructions
<https://pip.pypa.io/en/stable/installing/>`_ for details on how
to install ``pip``.
- Your installation of ``pip`` can be found in your ``PATH`` environment
variable. If it cannot, you may need to replace ``pip3`` in the commands
below with the full path to the executable.
- You are installing the crcmod package for use with your system installation
of Python, and thus use the ``sudo`` command. If installing crcmod for a
different Python environment (e.g. in a virtualenv), you should omit
``sudo`` from the commands below.
- You are using a Python 3 version with gsutil. You can determine which
Python version gsutil is using by running ``gsutil version -l`` and looking
for the ``python version: 2.x.x`` or ``python version: 3.x.x`` line.
CentOS, RHEL, and Fedora
------------------------
To compile and install crcmod:
yum install gcc python3-devel python3-setuptools redhat-rpm-config
sudo pip3 uninstall crcmod
sudo pip3 install --no-cache-dir -U crcmod
Debian and Ubuntu
-----------------
To compile and install crcmod:
sudo apt-get install gcc python3-dev python3-setuptools
sudo pip3 uninstall crcmod
sudo pip3 install --no-cache-dir -U crcmod
Enterprise SUSE
-----------------
To compile and install crcmod when using Enterprise SUSE for SAP 12:
sudo zypper install gcc python-devel
sudo pip uninstall crcmod
sudo pip install --no-cache-dir -U crcmod
To compile and install crcmod when using Enterprise SUSE for SAP 15:
sudo zypper install gcc python3-devel
sudo pip uninstall crcmod
sudo pip install --no-cache-dir -U crcmod
macOS
-----
gsutil distributes a pre-compiled version of crcmod for macOS, so you shouldn't
need to compile and install it yourself. If for some reason the pre-compiled
version is not being detected, please let the Google Cloud Storage team know
(see ``gsutil help support``).
To compile manually on macOS, you will first need to install
`Xcode <https://developer.apple.com/xcode/>`_ and then run:
pip3 install -U crcmod
Windows
-------
An installer is available for the compiled version of crcmod from the Python
Package Index (PyPi) at the following URL:
https://pypi.python.org/pypi/crcmod/1.7
NOTE: If you have installed crcmod and gsutil hasn't detected it, it may have
been installed to the wrong directory. It should be located at
<python_dir>\\files\\Lib\\site-packages\\crcmod\\
In some cases the installer will incorrectly install to
<python_dir>\\Lib\\site-packages\\crcmod\\
Manually copying the crcmod directory to the correct location should resolve
the issue.
""")
class CommandOptions(HelpProvider):
"""Additional help about CRC32C and installing crcmod."""
# Help specification. See help_provider.py for documentation.
help_spec = HelpProvider.HelpSpec(
help_name='crc32c',
help_name_aliases=['crc32', 'crc', 'crcmod'],
help_type='additional_help',
help_one_line_summary='CRC32C and Installing crcmod',
help_text=_DETAILED_HELP_TEXT,
subcommand_help_text={},
)

162
gcloud auth application-default login/google-cloud-sdk/platform/gsutil/gslib/addlhelp/creds.py Normal file
View File

@@ -0,0 +1,162 @@
# -*- coding: utf-8 -*-
# Copyright 2013 Google Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Additional help about types of credentials and authentication."""
from __future__ import absolute_import
from __future__ import print_function
from __future__ import division
from __future__ import unicode_literals
from gslib.help_provider import HelpProvider
_DETAILED_HELP_TEXT = ("""
<B>OVERVIEW</B>
gsutil currently supports several types of credentials/authentication, as
well as the ability to `access public data anonymously
<https://cloud.google.com/storage/docs/access-public-data>`_. Each of these
type of credentials is discussed in more detail below, along with
information about configuring and using credentials via the Cloud SDK.
<B>Configuring/Using Credentials via Cloud SDK Distribution of gsutil</B>
When gsutil is installed/used via the Cloud SDK ("gcloud"), credentials are
stored by Cloud SDK in a non-user-editable file located under
~/.config/gcloud (any manipulation of credentials should be done via the
gcloud auth command). If you need to set up multiple credentials (e.g., one
for an individual user account and a second for a service account), the
gcloud auth command manages the credentials for you, and you switch between
credentials using the gcloud auth command as well (for more details, see
https://cloud.google.com/sdk/gcloud/reference/auth).
Once credentials have been configured via ``gcloud auth``, those credentials
are used regardless of whether the user has any boto configuration files
(which are located at ~/.boto unless a different path is specified in the
BOTO_CONFIG environment variable). However, gsutil still looks for credentials
in the boto config file if a type of non-GCS credential is needed that's not
stored in the gcloud credential store (e.g., an HMAC credential for an S3
account).
<B>SUPPORTED CREDENTIAL TYPES</B>
gsutil supports several types of credentials (the specific subset depends on
which distribution of gsutil you are using; see above discussion).
OAuth2 user account:
This type of credential can be used for authenticating requests on behalf of
a specific user (which is probably the most common use of gsutil). This is
the default type of credential that is created when you run ``gcloud init``.
This credential type is not supported for stand-alone versions of gsutil.
For more details about OAuth2 authentication, see:
https://developers.google.com/accounts/docs/OAuth2#scenarios
HMAC:
This type of credential can be used by programs that are implemented using
HMAC authentication, which is an authentication mechanism supported by
certain other cloud storage service providers. This type of credential can
also be used for interactive use when moving data to/from service providers
that support HMAC credentials. This is the type of credential that is
created when you run ``gsutil config -a``.
Note that it's possible to set up HMAC credentials for both Google Cloud
Storage and another service provider; or to set up OAuth2 user account
credentials for Google Cloud Storage and HMAC credentials for another
service provider. To do so, after you run the ``gcloud init`` command, you
can edit the generated ~/.boto config file and look for comments for where
other credentials can be added.
For more details about HMAC authentication, see `HMAC keys for Cloud Storage
<https://cloud.google.com/storage/docs/authentication/hmackeys>`_.
OAuth2 service account:
This is a credential you can use when authenticating on
behalf of a service or application (as opposed to a user). For example, if
you intend to run gsutil out of a nightly cron job to upload/download data,
using a service account means the cron job does not depend on credentials of
an individual employee at your company. This is the type of credential that
is configured when you run ``gcloud auth login`` with the ``--cred-file``
flag (or ``gsutil config -e`` when using stand-alone versions of gsutil).
You should generally avoid this credential type because it requires storing
highly-privileged credentials in your local environment, creating potential
security risks. Instead, service account impersonation or workload identity
federation are the recommended alternatives when authenticating on behalf of
a service or application.
It is important to note that a service account is considered an Editor by
default for the purposes of API access, rather than an Owner. In particular,
the fact that Editors have OWNER access in the default object and
bucket ACLs, but the predefined ACL options remove OWNER access from
Editors, can lead to unexpected results. The solution to this problem is to
use "gsutil acl ch" instead of "gsutil acl set <predefined-ACL>" to change
permissions on a bucket.
To set up a service account for use with gsutil, see
`Authorize a service account using a service account key
<https://cloud.google.com/sdk/docs/authorizing#key>`_.
For more details about OAuth2 service accounts, see
`Using OAuth 2.0 for server to server applications
<https://developers.google.com/identity/protocols/oauth2/service-account>`_.
Compute Engine internal service account:
This is the type of service account used for accounts hosted by App Engine
or Compute Engine. Such credentials are created automatically for
you on Compute Engine when you run the ``gcloud compute instances create``
command and the credentials can be controlled with the ``--scopes`` flag.
For more details about using service account credentials for authenticating workloads
on Compute Engine, see
https://cloud.google.com/compute/docs/access/create-enable-service-accounts-for-instances.
For more details about App Engine service accounts, see
https://developers.google.com/appengine/docs/python/appidentity/overview
Service account impersonation:
Impersonating a service account is useful in scenarios where you need to
grant short-term access to specific resources. For example, if you have a
bucket of sensitive data that is typically read-only and want to
temporarily grant write access through a trusted service account.
You can specify which service account to use for impersonation by running
``gsutil -i``, ``gsutil config -e`` and editing the boto configuration file, or
``gcloud config set auth/impersonate_service_account [service_account_email_address]``.
In order to impersonate, your original credentials need to be granted
roles/iam.serviceAccountTokenCreator on the target service account.
For more information see
https://cloud.google.com/docs/authentication/use-service-account-impersonation.
External account credentials (workload identity federation):
Using workload identity federation, you can access Google Cloud resources
from Amazon Web Services (AWS), Microsoft Azure or any identity provider
that supports OpenID Connect (OIDC) or SAML 2.0.
For more information see
https://cloud.google.com/iam/docs/workload-identity-federation.
""")
class CommandOptions(HelpProvider):
"""Additional help about types of credentials and authentication."""
# Help specification. See help_provider.py for documentation.
help_spec = HelpProvider.HelpSpec(
help_name='creds',
help_name_aliases=['credentials', 'authentication', 'auth', 'gcloud'],
help_type='additional_help',
help_one_line_summary='Credential Types Supporting Various Use Cases',
help_text=_DETAILED_HELP_TEXT,
subcommand_help_text={},
)

149
gcloud auth application-default login/google-cloud-sdk/platform/gsutil/gslib/addlhelp/dev.py Normal file
View File

@@ -0,0 +1,149 @@
# -*- coding: utf-8 -*-
# Copyright 2012 Google Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Additional help about contributing code to gsutil."""
from __future__ import absolute_import
from __future__ import print_function
from __future__ import division
from __future__ import unicode_literals
from gslib.help_provider import HelpProvider
_DETAILED_HELP_TEXT = ("""
<B>OVERVIEW</B>
We're open to incorporating gsutil code changes authored by users. Here
are some guidelines:
1. Before we can accept code submissions, we have to jump a couple of legal
hurdles. Please fill out either the individual or corporate Contributor
License Agreement:
- If you are an individual writing original source code and you're
sure you own the intellectual property,
then you'll need to sign an individual CLA
(https://cla.developers.google.com/about/google-individual).
- If you work for a company that wants to allow you to contribute your
work to gsutil, then you'll need to sign a corporate CLA
(https://cla.developers.google.com/about/google-corporate)
Follow either of the two links above to access the appropriate CLA and
instructions for how to sign and return it. Once we receive it, we'll
add you to the official list of contributors and be able to accept
your patches.
2. If you found a bug or have an idea for a feature enhancement, we suggest
you check https://github.com/GoogleCloudPlatform/gsutil/issues to see if it
has already been reported by another user. From there you can also
subscribe to updates to the issue.
If a GitHub issue doesn't already exist, create one about your idea before
sending actual code. Often we can discuss the idea and help propose things
that could save you later revision work.
3. We tend to avoid adding command line options that are of use to only
a very small fraction of users, especially if there's some other way
to accommodate such needs. Adding such options complicates the code and
also adds overhead to users having to read through an "alphabet soup"
list of option documentation.
4. While gsutil has a number of features specific to Google Cloud Storage,
it can also be used with other cloud storage providers. We're open to
including changes for making gsutil support features specific to other
providers, as long as those changes don't make gsutil work worse for Google
Cloud Storage. If you do make such changes we recommend including someone
with knowledge of the specific provider as a code reviewer (see below).
5. You can check out the gsutil code from the GitHub repository:
https://github.com/GoogleCloudPlatform/gsutil
To clone a read-only copy of the repository:
git clone git://github.com/GoogleCloudPlatform/gsutil.git
To push your own changes to GitHub, click the Fork button on the
repository page and clone the repository from your own fork.
6. The gsutil git repository uses git submodules to pull in external modules.
After checking out the repository, make sure to also pull the submodules
by entering into the gsutil top-level directory and run:
git submodule update --init --recursive
7. Please make sure to run all tests against your modified code. To
do this, change directories into the gsutil top-level directory and run:
./gsutil test
The above tests take a long time to run because they send many requests to
the production service. The gsutil test command has a -u argument that will
only run unit tests. These run quickly, as they are executed with an
in-memory mock storage service implementation. To run only the unit tests,
run:
./gsutil test -u
If you made changes to boto, please run the boto tests. For these tests you
need to use HMAC credentials (from gsutil config -a), because the boto test
suite doesn't import the OAuth2 handler. You'll also need to install some
Python modules. Change directories into the boto root directory at
third_party/boto and run:
pip install -r requirements.txt
(You probably need to run this command using sudo.)
Make sure each of the individual installations succeeded. If they don't
you may need to run the install command again.
Then ensure your .boto file has HMAC credentials defined, and then change
directories into boto's tests directory and run:
python test.py unit
python test.py -t s3 -t gs -t ssl
8. Please consider contributing test code for your change, especially if the
change impacts any of the core gsutil code (like the gsutil cp command).
9. Please run the yapf linter with the config files in the root of the GitHub
repository:
yapf -irp .
10. When it's time to send us code, please submit a PR to the `gsutil GitHub
repository <https://github.com/GoogleCloudPlatform/gsutil>`_. For help on
making GitHub PRs, please refer to this
`GitHub help document <https://help.github.com/en/articles/about-pull-requests>`_.
""")
class CommandOptions(HelpProvider):
"""Additional help about contributing code to gsutil."""
# TODO: gsutil-beta: Add lint .rc file and linting instructions.
# Help specification. See help_provider.py for documentation.
help_spec = HelpProvider.HelpSpec(
help_name='dev',
help_name_aliases=[
'development',
'developer',
'code',
'mods',
'software',
],
help_type='additional_help',
help_one_line_summary='Contributing Code to gsutil',
help_text=_DETAILED_HELP_TEXT,
subcommand_help_text={},
)

173
gcloud auth application-default login/google-cloud-sdk/platform/gsutil/gslib/addlhelp/encoding.py Normal file
View File

@@ -0,0 +1,173 @@
# -*- coding: utf-8 -*-
# Copyright 2014 Google Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Additional help about CRC32C and installing crcmod."""
from __future__ import absolute_import
from __future__ import print_function
from __future__ import division
from __future__ import unicode_literals
from gslib.help_provider import HelpProvider
_DETAILED_HELP_TEXT = ("""
<B>OVERVIEW</B>
To reduce the chance for `filename encoding interoperability problems
<https://en.wikipedia.org/wiki/Filename#Encoding_indication_interoperability>`_
gsutil uses `UTF-8 <https://en.wikipedia.org/wiki/UTF-8>`_ character encoding
when uploading and downloading files. Because UTF-8 is in widespread (and
growing) use, for most users nothing needs to be done to use UTF-8. Users with
files stored in other encodings (such as
`Latin 1 <https://en.wikipedia.org/wiki/ISO/IEC_8859-1>`_) must convert those
filenames to UTF-8 before attempting to upload the files.
The most common place where users who have filenames that use some other
encoding encounter a gsutil error is while uploading files using the recursive
(-R) option on the gsutil cp , mv, or rsync commands. When this happens you'll
get an error like this:
CommandException: Invalid Unicode path encountered
('dir1/dir2/file_name_with_\\xf6n_bad_chars').
gsutil cannot proceed with such files present.
Please remove or rename this file and try again.
Note that the invalid Unicode characters have been hex-encoded in this error
message because otherwise trying to print them would result in another
error.
If you encounter such an error you can either remove the problematic file(s)
or try to rename them and re-run the command. If you have a modest number of
such files the simplest thing to do is to think of a different name for the
file and manually rename the file (using local filesystem tools). If you have
too many files for that to be practical, you can use a bulk rename tool or
script.
Unicode errors for valid Unicode filepaths can be caused by lack of Python
locale configuration on Linux and Mac OSes. If your file paths are Unicode
and you get encoding errors, ensure the LANG environment variable is set
correctly. Typically, the LANG variable should be set to something like
"en_US.UTF-8" or "de_DE.UTF-8".
Note also that there's no restriction on the character encoding used in file
content - it can be UTF-8, a different encoding, or non-character
data (like audio or video content). The gsutil UTF-8 character encoding
requirement applies only to filenames.
<B>USING UNICODE FILENAMES ON WINDOWS</B>
Windows support for Unicode in the command shell (cmd.exe or powershell) is
somewhat painful, because Windows uses a Windows-specific character encoding
called `cp1252 <https://en.wikipedia.org/wiki/Windows-1252>`_. To use Unicode
characters you need to run this command in the command shell before the first
time you use gsutil in that shell:
chcp 65001
If you neglect to do this before using gsutil, the progress messages while
uploading files with Unicode names or listing buckets with Unicode object
names will look garbled (i.e., with different glyphs than you expect in the
output). If you simply run the chcp command and re-run the gsutil command, the
output should no longer look garbled.
gsutil attempts to translate between cp1252 encoding and UTF-8 in the main
places that Unicode encoding/decoding problems have been encountered to date
(traversing the local file system while uploading files, and printing Unicode
names while listing buckets). However, because gsutil must perform
translation, it is likely there are other erroneous edge cases when using
Windows with Unicode. If you encounter problems, you might consider instead
using cygwin (on Windows) or Linux or macOS - all of which support Unicode.
<B>USING UNICODE FILENAMES ON MACOS</B>
macOS stores filenames in decomposed form (also known as
`NFD normalization <https://en.wikipedia.org/wiki/Unicode_equivalence>`_).
For example, if a filename contains an accented "e" character, that character
will be converted to an "e" followed by an accent before being saved to the
filesystem. As a consequence, it's possible to have different name strings
for files uploaded from an operating system that doesn't enforce decomposed
form (like Ubuntu) from one that does (like macOS).
The following example shows how this behavior could lead to unexpected
results. Say you create a file with non-ASCII characters on Ubuntu. Ubuntu
stores that filename in its composed form. When you upload the file to the
cloud, it is stored as named. But if you use gsutil rysnc to bring the file to
a macOS machine and edit the file, then when you use gsutil rsync to bring
this version back to the cloud, you end up with two different objects, instead
of replacing the original. This is because macOS converted the filename to
a decomposed form, and Cloud Storage sees this as a different object name.
<B>CROSS-PLATFORM ENCODING PROBLEMS OF WHICH TO BE AWARE</B>
Using UTF-8 for all object names and filenames will ensure that gsutil doesn't
encounter character encoding errors while operating on the files.
Unfortunately, it's still possible that files uploaded / downloaded this way
can have interoperability problems, for a number of reasons unrelated to
gsutil. For example:
- Windows filenames are case-insensitive, while Google Cloud Storage, Linux,
and macOS are not. Thus, for example, if you have two filenames on Linux
differing only in case and upload both to Google Cloud Storage and then
subsequently download them to Windows, you will end up with just one file
whose contents came from the last of these files to be written to the
filesystem.
- macOS performs character encoding decomposition based on tables stored in
the OS, and the tables change between Unicode versions. Thus the encoding
used by an external library may not match that performed by the OS. It is
possible that two object names may translate to a single local filename.
- Windows console support for Unicode is difficult to use correctly.
For a more thorough list of such issues see `this presentation
<http://www.i18nguy.com/unicode/filename-issues-iuc33.pdf>`_
These problems mostly arise when sharing data across platforms (e.g.,
uploading data from a Windows machine to Google Cloud Storage, and then
downloading from Google Cloud Storage to a machine running macOS).
Unfortunately these problems are a consequence of the lack of a filename
encoding standard, and users need to be aware of the kinds of problems that
can arise when copying filenames across platforms.
There is one precaution users can exercise to prevent some of these problems:
When using the Windows console specify wildcards or folders (using the -R
option) rather than explicitly named individual files.
<B>CONVERTING FILENAMES TO UNICODE</B>
Open-source tools are available to convert filenames for non-Unicode files.
For example, to convert from latin1 (a common Windows encoding) to Unicode,
you can use
`Windows iconv <http://gnuwin32.sourceforge.net/packages/libiconv.htm>`_.
For Unix-based systems, you can use
`libiconv <https://www.gnu.org/software/libiconv/>`_.
""")
class CommandOptions(HelpProvider):
"""Additional help about filename encoding and interoperability problems."""
# Help specification. See help_provider.py for documentation.
help_spec = HelpProvider.HelpSpec(
help_name='encoding',
help_name_aliases=[
'encodings',
'utf8',
'utf-8',
'latin1',
'unicode',
'interoperability',
],
help_type='additional_help',
help_one_line_summary='Filename encoding and interoperability problems',
help_text=_DETAILED_HELP_TEXT,
subcommand_help_text={},
)

103
gcloud auth application-default login/google-cloud-sdk/platform/gsutil/gslib/addlhelp/metadata.py Normal file
View File

@@ -0,0 +1,103 @@
# -*- coding: utf-8 -*-
# Copyright 2012 Google Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Additional help about object metadata."""
from __future__ import absolute_import
from __future__ import print_function
from __future__ import division
from __future__ import unicode_literals
from gslib.help_provider import HelpProvider
_DETAILED_HELP_TEXT = ("""
<B>OVERVIEW OF METADATA</B>
Objects can have associated metadata, which control aspects of how
GET requests are handled, including ``Content-Type``, ``Cache-Control``,
``Content-Disposition``, and ``Content-Encoding``. In addition, you can
set custom ``key:value`` metadata for use by your applications. For a
discussion of specific metadata properties, see the `metadata concept
page <https://cloud.google.com/storage/docs/metadata>`_.
There are two ways to set metadata on objects:
- At upload time you can specify one or more metadata properties to
associate with objects, using the ``gsutil -h option``. For example,
the following command would cause gsutil to set the ``Content-Type`` and
``Cache-Control`` for each of the files being uploaded from a local
directory named ``images``:
gsutil -h "Content-Type:text/html" \\
-h "Cache-Control:public, max-age=3600" cp -r images \\
gs://bucket/images
Note that -h is an option on the gsutil command, not the cp sub-command.
- You can set or remove metadata fields from already uploaded objects using
the ``gsutil setmeta`` command. See "gsutil help setmeta".
<B>SETTABLE FIELDS; FIELD VALUES</B>
You can't set some metadata fields, such as ETag and Content-Length. The
fields you can set are:
- ``Cache-Control``
- ``Content-Disposition``
- ``Content-Encoding``
- ``Content-Language``
- ``Content-Type``
- ``Custom-Time``
- Custom metadata
Field names are case-insensitive.
All fields and their values must consist only of ASCII characters, with the
exception of values for ``x-goog-meta-`` fields, which may contain arbitrary
Unicode values. Note that when setting metadata using the XML API, which sends
custom metadata as HTTP headers, Unicode characters are encoded using
UTF-8, then url-encoded to ASCII. For example:
gsutil setmeta -h "x-goog-meta-foo: ã" gs://bucket/object
stores the custom metadata key-value pair of ``foo`` and ``%C3%A3``.
Subsequently, running ``ls -L`` using the JSON API to list the object's
metadata prints ``%C3%A3``, while ``ls -L`` using the XML API
url-decodes this value automatically, printing the character ``ã``.
<B>VIEWING CURRENTLY SET METADATA</B>
You can see what metadata is currently set on an object by using:
gsutil ls -L gs://the_bucket/the_object
""")
class CommandOptions(HelpProvider):
"""Additional help about object metadata."""
# Help specification. See help_provider.py for documentation.
help_spec = HelpProvider.HelpSpec(
help_name='metadata',
help_name_aliases=[
'cache-control',
'caching',
'content type',
'mime type',
'mime',
'type',
],
help_type='additional_help',
help_one_line_summary='Working With Object Metadata',
help_text=_DETAILED_HELP_TEXT,
subcommand_help_text={},
)

188
gcloud auth application-default login/google-cloud-sdk/platform/gsutil/gslib/addlhelp/naming.py Normal file
View File

@@ -0,0 +1,188 @@
# -*- coding: utf-8 -*-
# Copyright 2012 Google Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Additional help about gsutil object and bucket naming."""
from __future__ import absolute_import
from __future__ import print_function
from __future__ import division
from __future__ import unicode_literals
from gslib.help_provider import HelpProvider
_DETAILED_HELP_TEXT = ("""
<B>BUCKET NAME REQUIREMENTS</B>
Google Cloud Storage has a single namespace, so you will not be allowed
to create a bucket with a name already in use by another user. You can,
however, carve out parts of the bucket name space corresponding to your
company's domain name (see "DOMAIN NAMED BUCKETS").
Bucket names must conform to standard DNS naming conventions. This is
because a bucket name can appear in a DNS record as part of a CNAME
redirect. In addition to meeting DNS naming requirements, Google Cloud
Storage imposes other requirements on bucket naming. At a minimum, your
bucket names must meet the following requirements:
- Bucket names must contain only lowercase letters, numbers, dashes (-), and
dots (.).
- Bucket names must start and end with a number or letter.
- Bucket names must contain 3 to 63 characters. Names containing dots can
contain up to 222 characters, but each dot-separated component can be
no longer than 63 characters.
- Bucket names cannot be represented as an IPv4 address in dotted-decimal
notation (for example, 192.168.5.4).
- Bucket names cannot begin with the "goog" prefix.
- For DNS compliance, you should not have a period adjacent to another
period or dash. For example, ".." or "-." or ".-" are not acceptable.
<B>OBJECT NAME REQUIREMENTS</B>
Object names can contain any sequence of Unicode characters, of length 1-1024
bytes when UTF-8 encoded. Object names must not contain CarriageReturn,
CarriageReturnLineFeed, or the XML-disallowed surrogate blocks (xFFFE
or xFFFF).
We strongly recommend that you abide by the following object naming
conventions:
- Avoid using control characters that are illegal in XML 1.0 in your object
names (#x7F-#x84 and #x86-#x9F). These characters will cause XML listing
issues when you try to list your objects.
- Avoid using "#" in your object names. gsutil interprets object names ending
with #<numeric string> as version identifiers, so including "#" in object
names can make it difficult or impossible to perform various operations on
such objects using gsutil (see 'gsutil help versions').
- Avoid using "[", "]", "*", or "?" in your object names. gsutil interprets
these characters as wildcards, so including any of these characters in
object names can make it difficult or impossible to perform various wildcard
operations using gsutil (see 'gsutil help wildcards').
See also 'gsutil help encoding' about file/object name encoding requirements
and potential interoperability concerns.
<B>DOMAIN NAMED BUCKETS</B>
You can carve out parts of the Google Cloud Storage bucket name space
by creating buckets with domain names (like "example.com").
Before you can create a bucket name containing one or more '.' characters,
the following rules apply:
- If the name is a syntactically valid DNS name ending with a
currently-recognized top-level domain (such as .com), you will be required
to verify domain ownership.
- Otherwise you will be disallowed from creating the bucket.
If your project needs to use a domain-named bucket, you need to have
a team member both verify the domain and create the bucket. This is
because Google Cloud Storage checks for domain ownership against the
user who creates the bucket, so the user who creates the bucket must
also be verified as an owner or manager of the domain.
To verify as the owner or manager of a domain, use the Google Webmaster
Tools verification process. The Webmaster Tools verification process
provides three methods for verifying an owner or manager of a domain:
1. Adding a special Meta tag to a site's homepage.
2. Uploading a special HTML file to a site.
3. Adding a DNS TXT record to a domain's DNS configuration.
Meta tag verification and HTML file verification are easier to perform and
are probably adequate for most situations. DNS TXT record verification is
a domain-based verification method that is useful in situations where a
site wants to tightly control who can create domain-named buckets. Once
a site creates a DNS TXT record to verify ownership of a domain, it takes
precedence over meta tag and HTML file verification. For example, you might
have two IT staff members who are responsible for managing your site, called
"example.com." If they complete the DNS TXT record verification, only they
would be able to create buckets called "example.com", "reports.example.com",
"downloads.example.com", and other domain-named buckets.
Site-Based Verification
-----------------------
If you have administrative control over the HTML files that make up a site,
you can use one of the site-based verification methods to verify that you
control or own a site. When you do this, Google Cloud Storage lets you
create buckets representing the verified site and any sub-sites - provided
nobody has used the DNS TXT record method to verify domain ownership of a
parent of the site.
As an example, assume that nobody has used the DNS TXT record method to verify
ownership of the following domains: abc.def.example.com, def.example.com,
and example.com. In this case, Google Cloud Storage lets you create a bucket
named abc.def.example.com if you verify that you own or control any of the
following sites:
http://abc.def.example.com
http://def.example.com
http://example.com
Domain-Based Verification
-------------------------
If you have administrative control over a domain's DNS configuration, you can
use the DNS TXT record verification method to verify that you own or control a
domain. When you use the domain-based verification method to verify that you
own or control a domain, Google Cloud Storage lets you create buckets that
represent any subdomain under the verified domain. Furthermore, Google Cloud
Storage prevents anybody else from creating buckets under that domain unless
you add their name to the list of verified domain owners or they have verified
their domain ownership by using the DNS TXT record verification method.
For example, if you use the DNS TXT record verification method to verify your
ownership of the domain example.com, Google Cloud Storage will let you create
bucket names that represent any subdomain under the example.com domain, such
as abc.def.example.com, example.com/music/jazz, or abc.example.com/music/jazz.
Using the DNS TXT record method to verify domain ownership supersedes
verification by site-based verification methods. For example, if you
use the Meta tag method or HTML file method to verify domain ownership
of http://example.com, but someone else uses the DNS TXT record method
to verify ownership of the example.com domain, Google Cloud Storage will
not allow you to create a bucket named example.com. To create the bucket
example.com, the domain owner who used the DNS TXT method to verify domain
ownership must add you to the list of verified domain owners for example.com.
The DNS TXT record verification method is particularly useful if you manage
a domain for a large organization that has numerous subdomains because it
lets you control who can create buckets representing those domain names.
Note: If you use the DNS TXT record verification method to verify ownership of
a domain, you cannot create a CNAME record for that domain. RFC 1034 disallows
inclusion of any other resource records if there is a CNAME resource record
present. If you want to create a CNAME resource record for a domain, you must
use the Meta tag verification method or the HTML file verification method.
""")
class CommandOptions(HelpProvider):
"""Additional help about gsutil object and bucket naming."""
# Help specification. See help_provider.py for documentation.
help_spec = HelpProvider.HelpSpec(
help_name='naming',
help_name_aliases=['domain', 'limits', 'name', 'names'],
help_type='additional_help',
help_one_line_summary='Object and Bucket Naming',
help_text=_DETAILED_HELP_TEXT,
subcommand_help_text={},
)

160
gcloud auth application-default login/google-cloud-sdk/platform/gsutil/gslib/addlhelp/prod.py Normal file
View File

@@ -0,0 +1,160 @@
# -*- coding: utf-8 -*-
# Copyright 2012 Google Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Additional help about using gsutil for production tasks."""
from __future__ import absolute_import
from __future__ import print_function
from __future__ import division
from __future__ import unicode_literals
from gslib.help_provider import HelpProvider
from gslib.utils.constants import RESUMABLE_THRESHOLD_MIB
_DETAILED_HELP_TEXT = ("""
<B>OVERVIEW</B>
If you use gsutil in large production tasks (such as uploading or
downloading many GiBs of data each night), there are a number of things
you can do to help ensure success. Specifically, this section discusses
how to script large production tasks around gsutil's resumable transfer
mechanism.
<B>BACKGROUND ON RESUMABLE TRANSFERS</B>
First, it's helpful to understand gsutil's resumable transfer mechanism,
and how your script needs to be implemented around this mechanism to work
reliably. gsutil uses resumable transfer support when you attempt to download
a file of any size or to upload a file larger than a configurable threshold
(by default, this threshold is %d MiB). If a transfer fails partway through
(e.g., because of an intermittent network problem), gsutil uses a
`truncated randomized binary exponential backoff-and-retry strategy
<https://cloud.google.com/storage/docs/retry-strategy#tools>`_ that by
default retries transfers up to 23 times over a 10 minute period of time. If
the transfer fails each of these attempts with no intervening progress,
gsutil gives up on the transfer, but keeps a "tracker" file for it in a
configurable location (the default location is ~/.gsutil/, in a file named
by a combination of the SHA1 hash of the name of the bucket and object being
transferred and the last 16 characters of the file name). When transfers
fail in this fashion, you can rerun gsutil at some later time (e.g., after
the networking problem has been resolved), and the resumable transfer picks
up where it left off.
<B>SCRIPTING DATA TRANSFER TASKS</B>
To script large production data transfer tasks around this mechanism,
you can implement a script that runs periodically, determines which file
transfers have not yet succeeded, and runs gsutil to copy them. Below,
we offer a number of suggestions about how this type of scripting should
be implemented:
1. When resumable transfers fail without any progress 23 times in a row
over the course of up to 10 minutes, it probably won't work to simply
retry the transfer immediately. A more successful strategy would be to
have a cron job that runs every 30 minutes, determines which transfers
need to be run, and runs them. If the network experiences intermittent
problems, the script picks up where it left off and will eventually
succeed (once the network problem has been resolved).
2. If your business depends on timely data transfer, you should consider
implementing some network monitoring. For example, you can implement
a task that attempts a small download every few minutes and raises an
alert if the attempt fails for several attempts in a row (or more or less
frequently depending on your requirements), so that your IT staff can
investigate problems promptly. As usual with monitoring implementations,
you should experiment with the alerting thresholds, to avoid false
positive alerts that cause your staff to begin ignoring the alerts.
3. There are a variety of ways you can determine what files remain to be
transferred. We recommend that you avoid attempting to get a complete
listing of a bucket containing many objects (e.g., tens of thousands
or more). One strategy is to structure your object names in a way that
represents your transfer process, and use gsutil prefix wildcards to
request partial bucket listings. For example, if your periodic process
involves downloading the current day's objects, you could name objects
using a year-month-day-object-ID format and then find today's objects by
using a command like gsutil ls "gs://bucket/2011-09-27-*". Note that it
is more efficient to have a non-wildcard prefix like this than to use
something like gsutil ls "gs://bucket/*-2011-09-27". The latter command
actually requests a complete bucket listing and then filters in gsutil,
while the former asks Google Storage to return the subset of objects
whose names start with everything up to the "*".
For data uploads, another technique would be to move local files from a "to
be processed" area to a "done" area as your script successfully copies
files to the cloud. You can do this in parallel batches by using a command
like:
gsutil -m cp -r to_upload/subdir_$i gs://bucket/subdir_$i
where i is a shell loop variable. Make sure to check the shell $status
variable is 0 after each gsutil cp command, to detect if some of the copies
failed, and rerun the affected copies.
With this strategy, the file system keeps track of all remaining work to
be done.
4. If you have really large numbers of objects in a single bucket
(say hundreds of thousands or more), you should consider tracking your
objects in a database instead of using bucket listings to enumerate
the objects. For example this database could track the state of your
downloads, so you can determine what objects need to be downloaded by
your periodic download script by querying the database locally instead
of performing a bucket listing.
5. Make sure you don't delete partially downloaded temporary files after a
transfer fails: gsutil picks up where it left off (and performs a hash
of the final downloaded content to ensure data integrity), so deleting
partially transferred files will cause you to lose progress and make
more wasteful use of your network.
6. If you have a fast network connection, you can speed up the transfer of
large numbers of files by using the gsutil -m (multi-threading /
multi-processing) option. Be aware, however, that gsutil doesn't attempt to
keep track of which files were downloaded successfully in cases where some
files failed to download. For example, if you use multi-threaded transfers
to download 100 files and 3 failed to download, it is up to your scripting
process to determine which transfers didn't succeed, and retry them. A
periodic check-and-run approach like outlined earlier would handle this
case.
If you use parallel transfers (gsutil -m) you might want to experiment with
the number of threads being used (via the parallel_thread_count setting
in the .boto config file). By default, gsutil uses 10 threads for Linux
and 24 threads for other operating systems. Depending on your network
speed, available memory, CPU load, and other conditions, this may or may
not be optimal. Try experimenting with higher or lower numbers of threads
to find the best number of threads for your environment.
""" % RESUMABLE_THRESHOLD_MIB)
class CommandOptions(HelpProvider):
"""Additional help about using gsutil for production tasks."""
# Help specification. See help_provider.py for documentation.
help_spec = HelpProvider.HelpSpec(
help_name='prod',
help_name_aliases=[
'production',
'resumable',
'resumable upload',
'resumable transfer',
'resumable download',
'scripts',
'scripting',
],
help_type='additional_help',
help_one_line_summary='Scripting Production Transfers',
help_text=_DETAILED_HELP_TEXT,
subcommand_help_text={},
)

221
gcloud auth application-default login/google-cloud-sdk/platform/gsutil/gslib/addlhelp/security.py Normal file
View File

@@ -0,0 +1,221 @@
# -*- coding: utf-8 -*-
# Copyright 2015 Google Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Additional help about types of credentials and authentication."""
from __future__ import absolute_import
from __future__ import print_function
from __future__ import division
from __future__ import unicode_literals
from gslib.help_provider import HelpProvider
_DETAILED_HELP_TEXT = ("""
<B>OVERVIEW</B>
This help section provides details about various precautions taken by gsutil
to protect data security, as well as recommendations for how customers should
safeguard security.
<B>TRANSPORT LAYER SECURITY</B>
gsutil performs all operations using transport-layer encryption (HTTPS), to
protect against data leakage over shared network links. This is also important
because gsutil uses "bearer tokens" for authentication (OAuth2) as well as for
resumable upload identifiers, and such tokens must be protected from being
eavesdropped and reused.
gsutil also supports the older HMAC style of authentication via the XML API
(see `gsutil endpoints
<https://cloud.google.com/storage/docs/request-endpoints#gsutil>`_). While
HMAC authentication does not use bearer tokens (and thus is not subject to
eavesdropping/replay attacks), it's still important to encrypt data traffic.
To add an extra layer of security, gsutil supports mutual TLS (mTLS) for
the Cloud Storage JSON API. With mTLS, the client verifies the server
certificate, and the server also verifies the client.
To find out more about how to enable mTLS, see the `install docs
<https://cloud.google.com/storage/docs/gsutil_install>`_.
<B>LOCAL FILE STORAGE SECURITY</B>
gsutil takes a number of precautions to protect against security exploits in
the files it stores locally:
- When the ``gcloud init``, ``gsutil config -a``, or ``gsutil config -e``
commands run, they set file protection mode 600 ("-rw-------") on the .boto
configuration file they generate, so only the user (or superuser) can read
it. This is important because these files contain security-sensitive
information, including credentials and proxy configuration.
- These commands also use file protection mode 600 for the private key file
stored locally when you create service account credentials.
- The default level of logging output from gsutil commands does not include
security-sensitive information, such as OAuth2 tokens and proxy
configuration information. (See the "RECOMMENDED USER PRECAUTIONS" section
below if you increase the level of debug output, using the gsutil -D
option.)
Note that protection modes are not supported on Windows, so if you
use gsutil on Windows we recommend using an encrypted file system and strong
account passwords.
<B>SECURITY-SENSITIVE FILES WRITTEN TEMPORARILY TO DISK BY GSUTIL</B>
gsutil buffers data in temporary files in several situations:
- While compressing data being uploaded via gsutil cp -z/-Z, gsutil
buffers the data in temporary files with protection 600, which it
deletes after the upload is complete (similarly for downloading files
that were uploaded with gsutil cp -z/-Z or some other process that sets the
Content-Encoding to "gzip"). However, if you kill the gsutil process
while the upload is under way the partially written file will be left
in place. See the "CHANGING TEMP DIRECTORIES" section in
"gsutil help cp" for details of where the temporary files are written
and how to change the temp directory location.
- When performing a resumable upload gsutil stores the upload ID (which,
as noted above, is a bearer token and thus should be safe-guarded) in a
file under ~/.gsutil/tracker-files with protection 600, and deletes this
file after the upload is complete. However, if the upload doesn't
complete successfully the tracker file is left in place so the resumable
upload can be re-attempted later. Over time it's possible to accumulate
these tracker files from aborted upload attempts, though resumable
upload IDs are only valid for 1 week, so the security risk only exists
for files less than that age. If you consider the risk of leaving
aborted upload IDs in the tracker directory too high you could modify
your upload scripts to delete the tracker files; or you could create a
cron job to clear the tracker directory periodically.
- The gsutil rsync command stores temporary files (with protection 600)
containing the names, sizes, and checksums of source and destination
directories/buckets, which it deletes after the rsync is complete.
However, if you kill the gsutil process while the rsync is under way the
listing files will be left in place.
Note that gsutil deletes temporary files using the standard OS unlink system
call, which does not perform `data wiping
<https://en.wikipedia.org/wiki/Data_erasure>`_. Thus, the content of such
temporary files can be recovered by a determined adversary.
<B>ACCESS CONTROL LISTS</B>
Unless you specify a different ACL (e.g., via the gsutil cp -a option), by
default objects written to a bucket use the default object ACL on that bucket.
Unless you modify that ACL (e.g., via the gsutil defacl command), by default
it will allow all project editors write access to the object and read/write
access to the object's metadata and will allow all project viewers read
access to the object.
The Google Cloud Storage access control system includes the ability to
specify that objects are publicly readable. Make sure you intend for any
objects you write with this permission to be public. Once "published", data
on the Internet can be copied to many places, so it's effectively impossible
to regain read control over an object written with this permission.
The Google Cloud Storage access control system includes the ability to
specify that buckets are publicly writable. While configuring a bucket this
way can be convenient for various purposes, we recommend against using this
permission - it can be abused for distributing illegal content, viruses, and
other malware, and the bucket owner is legally and financially responsible
for the content stored in their buckets. If you need to make content
available to customers who don't have Google accounts consider instead using
signed URLs (see "gsutil help signurl").
<B>SOFTWARE INTEGRITY AND UPDATES</B>
gsutil is distributed as a part of the bundled Cloud SDK release. This
distribution method takes a variety of security precautions to protect the
integrity of the software. We strongly recommend against getting a copy of
gsutil from any other sources (such as mirror sites).
<B>PROXY USAGE</B>
gsutil supports access via proxies, such as Squid and a number of commercial
products. A full description of their capabilities is beyond the scope of this
documentation, but proxies can be configured to support many security-related
functions, including virus scanning, Data Leakage Prevention, control over
which certificates/CA's are trusted, content type filtering, and many more
capabilities. Some of these features can slow or block legitimate gsutil
behavior. For example, virus scanning depends on decrypting file content,
which in turn requires that the proxy terminate the gsutil connection and
establish a new connection - and in some cases proxies will rewrite content in
ways that result in checksum validation errors and other problems.
For details on configuring proxies, see the proxy help text generated in your
.boto configuration file by the ``gcloud init``, ``gsutil -a``, and
``gsutil -e`` commands.
<B>ENCRYPTION AT REST</B>
All Google Cloud Storage data are automatically stored in an encrypted state,
but you can also provide your own encryption keys. For more information, see
`Cloud Storage Encryption
<https://cloud.google.com/storage/docs/encryption>`_.
<B>DATA PRIVACY</B>
Google will never ask you to share your credentials, password, or other
security-sensitive information. Beware of potential phishing scams where
someone attempts to impersonate Google and asks for such information.
<B>MEASUREMENT DATA</B>
The gsutil perfdiag command collects a variety of performance-related
measurements and details about your local system and network environment, for
use in troubleshooting performance problems. None of this information will be
sent to Google unless you choose to send it.
<B>RECOMMENDED USER PRECAUTIONS</B>
The first and foremost precaution is: Never share your credentials. Each user
should have distinct credentials.
If you run gsutil -D (to generate debugging output) it will include OAuth2
refresh and access tokens in the output. Make sure to redact this information
before sending this debug output to anyone during troubleshooting/tech support
interactions.
If you run gsutil --trace-token (to send a trace directly to Google),
sensitive information like OAuth2 tokens and the contents of any files
accessed during the trace may be included in the content of the trace.
Customer-supplied encryption key information in the .boto configuration is
security sensitive.
The proxy configuration information in the .boto configuration is
security-sensitive, especially if your proxy setup requires user and
password information. Even if your proxy setup doesn't require user and
password, the host and port number for your proxy is often considered
security-sensitive. Protect access to your .boto configuration file.
If you are using gsutil from a production environment (e.g., via a cron job
running on a host in your data center), use service account credentials rather
than individual user account credentials. These credentials were designed for
such use and, for example, protect you from losing access when an employee
leaves your company.
""")
class CommandOptions(HelpProvider):
"""Additional help about security and privacy considerations using gsutil."""
# Help specification. See help_provider.py for documentation.
help_spec = HelpProvider.HelpSpec(
help_name='security',
help_name_aliases=['protection', 'privacy', 'proxies', 'proxy'],
help_type='additional_help',
help_one_line_summary='Security and Privacy Considerations',
help_text=_DETAILED_HELP_TEXT,
subcommand_help_text={},
)

308
gcloud auth application-default login/google-cloud-sdk/platform/gsutil/gslib/addlhelp/shim.py Normal file
View File

@@ -0,0 +1,308 @@
# -*- coding: utf-8 -*-
# Copyright 2013 Google Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Additional help about the shim for running gcloud storage."""
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
from __future__ import unicode_literals
from gslib.help_provider import HelpProvider
_DETAILED_HELP_TEXT = ("""
<B>OVERVIEW</B>
Cloud SDK includes a new CLI, gcloud storage, that can be considerably faster
than gsutil when performing uploads and downloads with less parameter
tweaking. This new CLI has a syntax and command structure that is familiar to
gsutil users but is fundamentally different in many important ways. To ease
transition to this new CLI, gsutil provides a shim that translates your gsutil
commands to gcloud storage commands if an equivalent exists, and falls back to
gsutil's usual behavior if an equivalent does not exist.
<B> PREREQUISITES </B>
You should be running gsutil binary installed via Cloud SDK tool.
To verify this, run ``gsutil ver -l`` and you should be able to see
``pass cloud sdk credentials to gsutil: True``. If it is ``False``,
try running ``gcloud config set pass_credentials_to_gsutil True``.
If you do not have ``gcloud`` installed, follow the
`Cloud SDK installation instructions. <https://cloud.google.com/sdk/docs/install>`_.
<B>TO ENABLE</B>
Set ``use_gcloud_storage=True`` in the ``.boto`` config file under the
``[GSUtil]`` section:
[GSUtil]
use_gcloud_storage=True
You can also set the flag for individual commands using the top-level ``-o``
flag:
gsutil -o "GSUtil:use_gcloud_storage=True" -m cp -p file gs://bucket/obj
<B>AVAILABLE COMMANDS</B>
The gcloud storage CLI only supports a subset of gsutil commands. What follows
is a list of commands supported by the shim with any differences in behavior
noted.
acl
------------------------
- The ``ch`` subcommand is not supported.
autoclass
------------------------
- Works as expected.
bucketpolicyonly
------------------------
- Works as expected.
cat
------------------------
- Prints object data for a second object even if the first object is invalid.
compose
------------------------
- Works as expected.
cors
------------------------
- ``get`` subcommand prints "[]" instead of "gs://[bucket name] has no CORS
configuration".
cp
------------------------
- Copies a second object even if the first object is invalid.
- Does not support file to file copies.
- Supports copying objects cloud-to-cloud with trailing slashes in the name.
- The all-version flag (``-A``) silently enables sequential execution rather
than raising an error.
defacl
------------------------
- The ``ch`` subcommand is not supported.
defstorageclass
------------------------
- Works as expected.
hash
------------------------
- In gsutil, the ``-m`` and ``-c`` flags that affect which hashes are displayed
are ignored for cloud objects. This behavior is fixed for the shim and gcloud
storage.
iam
------------------------
- The ``ch`` subcommand is not supported.
- The ``-f`` flag will continue on any error, not just API errors.
kms
------------------------
- The authorize subcommand returns informational messages in a different
format.
- The encryption subcommand returns informational messages in a different
format.
labels
------------------------
- ``get`` subcommand prints "[]" instead of "gs://[bucket name] has no labels
configuration."
lifecycle
------------------------
- Works as expected.
logging
------------------------
- The get subcommand has different JSON spacing and doesn't print an
informational message if no configuration is found.
ls
------------------------
- `ls -L` output uses spaces instead of tabs.
mb
------------------------
- Works as expected.
mv
------------------------
- See notes on cp.
notification
------------------------
- The list subcommand prints configuration information as YAML.
- The delete subcommand offers progress tracking and parallelization.
pap
------------------------
- Works as expected.
rb
------------------------
- Works as expected.
requesterpays
------------------------
- Works as expected.
rewrite
------------------------
- The -k flag does not throw an error if called without a new key. In both the
shim and unshimmed cases, the old key is maintained.
rm
------------------------
- ``$folder$`` delete markers are not supported.
rpo
------------------------
- Works as expected.
setmeta
------------------------
- Does not throw an error if no headers are changed.
stat
------------------------
- Includes a field "Storage class update time:" which may throw off tabbing.
ubla
------------------------
- Works as expected.
versioning
------------------------
- Works as expected.
web
------------------------
- The get subcommand has different JSON spacing and doesn't print an
informational message if no configuration is found.
<B>BOTO CONFIGURATION</B>
Configuration found in the boto file is mapped 1:1 to gcloud environment
variables where appropriate.
Important: The ``gs_oauth2_refresh_token`` and ``gs_service_key_file`` fields
are ignored by the shim and authentication is taken care by ``gcloud``.
[Credentials]
------------------------
- aws_access_key_id: AWS_ACCESS_KEY_ID
- aws_secret_access_key: AWS_SECRET_ACCESS_KEY
- use_client_certificate: CLOUDSDK_CONTEXT_AWARE_USE_CLIENT_CERTIFICATE
[Boto]
------------------------
- proxy: CLOUDSDK_PROXY_ADDRESS
- proxy_type: CLOUDSDK_PROXY_TYPE
- proxy_port: CLOUDSDK_PROXY_PORT
- proxy_user: CLOUDSDK_PROXY_USERNAME
- proxy_pass: CLOUDSDK_PROXY_PASSWORD
- proxy_rdns: CLOUDSDK_PROXY_RDNS
- http_socket_timeout: CLOUDSDK_CORE_HTTP_TIMEOUT
- ca_certificates_file: CLOUDSDK_CORE_CUSTOM_CA_CERTS_FILE
- max_retry_delay: CLOUDSDK_STORAGE_BASE_RETRY_DELAY
- num_retries: CLOUDSDK_STORAGE_MAX_RETRIES
[GSUtil]
------------------------
- check_hashes: CLOUDSDK_STORAGE_CHECK_HASHES
- default_project_id: CLOUDSDK_CORE_PROJECT
- disable_analytics_prompt: CLOUDSDK_CORE_DISABLE_USAGE_REPORTING
- use_magicfile: CLOUDSDK_STORAGE_USE_MAGICFILE
- parallel_composite_upload_threshold: CLOUDSDK_STORAGE_PARALLEL_COMPOSITE_UPLOAD_THRESHOLD
- resumable_threshold: CLOUDSDK_STORAGE_RESUMABLE_THRESHOLD
[OAuth2]
------------------------
- client_id: CLOUDSDK_AUTH_CLIENT_ID
- client_secret: CLOUDSDK_AUTH_CLIENT_SECRET
- provider_authorization_uri: CLOUDSDK_AUTH_AUTH_HOST
- provider_token_uri: CLOUDSDK_AUTH_TOKEN_HOST
<B>GENERAL COMPATIBILITY NOTES</B>
- Due to its compatibility across all major platforms, multiprocessing is
enabled for all commands by default (equivalent to the -m option always
being included in gsutil).
- A sequence of asterisks greater than 2 (i.e. ``***``) are always treated as
a single asterisk.
- Unlike gsutil, gcloud is not designed to be used in parallel invocations,
and doing so (i.e. running the shim from 2 terminals at once) can lead to
unpredictable behavior.
- Assuming a bucket contains an object ``gs://bucket/nested/foo.txt``,
gsutil's wildcard iterator will match ``foo.txt`` given a URL like
``gs://bucket/*/nested/*``. The shim will not match ``foo.txt`` given the
same URL.
- This will be updated as new commands are supported by both gcloud storage
and the shim.
- If Unicode is having issues, try setting the environment variable
``PYTHONUTF8`` to ``1``. Specifically, this may help on the Windows
command-line (CMD).
""")
class CommandOptions(HelpProvider):
"""Additional help about the shim for running gcloud storage."""
# Help specification. See help_provider.py for documentation.
help_spec = HelpProvider.HelpSpec(
help_name='shim',
help_name_aliases=['gcloudstorage', 'gcloud storage'],
help_type='additional_help',
help_one_line_summary='Shim for Running gcloud storage',
help_text=_DETAILED_HELP_TEXT,
subcommand_help_text={},
)

104
gcloud auth application-default login/google-cloud-sdk/platform/gsutil/gslib/addlhelp/support.py Normal file
View File

@@ -0,0 +1,104 @@
# -*- coding: utf-8 -*-
# Copyright 2012 Google Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Additional help about technical and billing support."""
from __future__ import absolute_import
from __future__ import print_function
from __future__ import division
from __future__ import unicode_literals
from gslib.help_provider import HelpProvider
_DETAILED_HELP_TEXT = ("""
<B>TECHNICAL SUPPORT</B>
If you have any questions or encounter any problems with Google Cloud Storage,
please first read the `FAQ <https://cloud.google.com/storage/docs/faq>`_.
If you still have questions please use one of the following methods as
appropriate, providing the details noted below:
A) For API, tool usage, or other software development-related questions,
please search for and post questions on Stack Overflow, using the official
`google-cloud-storage tag
<http://stackoverflow.com/questions/tagged/google-cloud-storage>`_. Our
support team actively monitors questions to this tag and we'll do our best to
respond.
B) For gsutil bugs or feature requests, please check if there is already a
`existing GitHub issue <https://github.com/GoogleCloudPlatform/gsutil/issues>`_
that covers your request. If not, create a
`new GitHub issue <https://github.com/GoogleCloudPlatform/gsutil/issues/new>`_.
To help us diagnose any issues you encounter, when creating a new issue
please provide these details in addition to the description of your problem:
- The resource you are attempting to access (bucket name, object name),
assuming they are not sensitive.
- The operation you attempted (GET, PUT, etc.)
- The time and date (including timezone) at which you encountered the problem
- If you can use gsutil to reproduce your issue, specify the -D option to
display your request's HTTP details, and provide these details in the
issue.
Warning: The gsutil -d, -D, and -DD options will also print the authentication
header with authentication credentials for your Google Cloud Storage account.
Make sure to remove any "Authorization:" headers before you post HTTP details
to the issue. Note also that if you upload files large enough to use resumable
uploads, the resumable upload IDs are security-sensitive while an upload
is not yet complete, so should not be posted on public forums.
If you make any local modifications to gsutil, please make sure to use
a released copy of gsutil (instead of your locally modified copy) when
providing the gsutil -D output noted above. We cannot support versions
of gsutil that include local modifications. (However, we're open to user
contributions; see "gsutil help dev".)
<B>BILLING AND ACCOUNT QUESTIONS</B>
A) For billing documentation, please visit
https://cloud.google.com/storage/pricing.
If you want to cancel billing, follow the instructions at
`Cloud Storage FAQ <https://cloud.google.com/storage/docs/faq#disablebilling>`_.
Caution: When you disable billing, you also disable the Google Cloud Storage
service. Make sure you want to disable the Google Cloud Storage service
before you disable billing.
B) For support regarding billing, please see
`billing support <https://support.google.com/cloud/contact/cloud_platform_billing>`_.
For other questions regarding your account, Terms Of Service, Google
Cloud Console, or other administration-related questions please see
`Google Cloud Platform support <https://support.google.com/cloud/answer/6282346#gcp>`_.
""")
class CommandOptions(HelpProvider):
"""Additional help about technical and billing support."""
# Help specification. See help_provider.py for documentation.
help_spec = HelpProvider.HelpSpec(
help_name='support',
help_name_aliases=[
'techsupport',
'tech support',
'technical support',
'billing',
'faq',
'questions',
],
help_type='additional_help',
help_one_line_summary='Google Cloud Storage Support',
help_text=_DETAILED_HELP_TEXT,
subcommand_help_text={},
)

87
gcloud auth application-default login/google-cloud-sdk/platform/gsutil/gslib/addlhelp/versions.py Normal file
View File

@@ -0,0 +1,87 @@
# -*- coding: utf-8 -*-
# Copyright 2012 Google Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Additional help about object versioning."""
from __future__ import absolute_import
from __future__ import print_function
from __future__ import division
from __future__ import unicode_literals
from gslib.help_provider import HelpProvider
_DETAILED_HELP_TEXT = ("""
<B>OVERVIEW</B>
Versioning-enabled buckets maintain noncurrent versions of objects, providing
a way to un-delete data that you accidentally deleted, or to retrieve older
versions of your data. Noncurrent objects are ignored by gsutil commands
unless you indicate it should do otherwise by setting a relevant command flag
or by including a specific generation number in your command. For example,
wildcards like ``*`` and ``**`` do not, by themselves, act on noncurrent
object versions.
When using gsutil cp, you cannot specify a version-specific URL as the
destination, because writes to Cloud Storage always create a new version.
Trying to specify a version-specific URL as the destination of ``gsutil cp``
results in an error. When you specify a noncurrent object as a source in a
copy command, you always create a new object version and retain the original
(even when using the command to restore a live version). You can use the
``gsutil mv`` command to simultaneously restore an object version and remove
the noncurrent copy that was used as the source.
You can turn versioning on or off for a bucket at any time. Turning
versioning off leaves existing object versions in place and simply causes
the bucket to delete the existing live version of the object whenever a new
version is uploaded.
Regardless of whether you have enabled versioning on a bucket, every object
has two associated positive integer fields:
- the generation, which is updated when a new object replaces an existing
object with the same name. Note that there is no guarantee that generation
numbers increase for successive versions, only that each new version has a
unique generation number.
- the metageneration, which identifies the metadata generation. It starts
at 1; is updated every time the metadata (e.g., ACL or Content-Type) for a
given content generation is updated; and gets reset when the generation
number changes.
Of these two integers, only the generation is used when working with versioned
data. Both generation and metageneration can be used with concurrency control.
To learn more about versioning and concurrency, see the following documentation:
- `Overview of Object Versioning
<https://cloud.google.com/storage/docs/object-versioning>`_
- `Guide for using Object Versioning
<https://cloud.google.com/storage/docs/using-object-versioning>`_
- The `reference page for the gsutil versioning command
<https://cloud.google.com/storage/docs/gsutil/commands/versioning>`_
- `Request preconditions
<https://cloud.google.com/storage/docs/request-preconditions>`_
""")
class CommandOptions(HelpProvider):
"""Additional help about object versioning."""
# Help specification. See help_provider.py for documentation.
help_spec = HelpProvider.HelpSpec(
help_name='versions',
help_name_aliases=['concurrency', 'concurrency control'],
help_type='additional_help',
help_one_line_summary='Object Versioning and Concurrency Control',
help_text=_DETAILED_HELP_TEXT,
subcommand_help_text={},
)