How to Upload Large Size Folders to Github

goodls

Overview

This is a CLI tool to download shared files and folders from Google Drive. For large file, the resumable download can be also run.

Methods

1. Download shared files from the shared URL without the authorization.

We accept already known that the shared files on Google Drive can exist downloaded without the authorization. Merely when the size of file becomes large (virtually 40MB), it requires a little ingenuity to download the file. It requires to admission 2 times to Google Drive. At 1st access, it retrieves a cookie and a lawmaking for downloading. At 2nd admission, the file is downloaded using the cookie and code. I created this process as a CLI tool.

2. Download all shared files with the folder structure from the shared folder.

This method uses API central.

There are sometimes the situation for downloading files in a shared binder. But I couldn't find the CLI applications for downloading files in the shared folder. So I implemented this. But when in lodge to retrieve the file list from the shared file, Bulldoze API is required to be used. In gild to use Drive API, it is required to utilize OAuth2, Service account and API key. And so I selected to use API key which is the simplest mode. This CLI tool can retrieve the file list in the shared folder using API key and download all files in the shared binder. From version 1.2.2, you can retrieve only files with the specific mimeType from the folder.

3. Run resumable download for large files.

This method uses API cardinal.

At a recent proposal, I knew the requirement of the resumable download of shared file. And then I implemented this.

How to Install

Download an executable file of goodls from the release page and import to a directory with path.

or

Use go get.

$ get install github.com/tanaikech/goodls@latest

• GO111MODULE=on

Usage

1. Download shared files

The paradigm used for this sit-in was created by k3-studio

You can use this simply afterward yous download or install goodls. You are not required to practise similar OAuth2 procedure.

$ goodls -u [URL of shared file on Google Drive]

• Help
  • $ goodls --help
• Options
  • -e
    • Extension of output file. This is for simply Google Docs (Spreadsheet, Document, Presentation). Default is pdf. When ms is used, the shared Google Docs can be downloaded every bit Microsoft Docs.
    • Sample :
      • $ goodls -u https://docs.google.com/document/d/#####/edit?usp=sharing -e txt
  • -f
    • Filename of file which is output. When this was not used, the original filename on Google Drive is used.
    • Sample :
      • $ goodls -u https://docs.google.com/document/d/#####/edit?usp=sharing -e txt -f sample.txt
• URL is similar beneath.
  • In the example of Google Docs (Spreadsheet, Document, Slides)
    • https://docs.google.com/spreadsheets/d/#####/edit?usp=sharing
    • https://docs.google.com/document/d/#####/edit?usp=sharing
    • https://docs.google.com/presentation/d/#####/edit?usp=sharing
  • In the case of except for Google Docs
    • https://bulldoze.google.com/file/d/#####/view?usp=sharing
  • In the case of webContentLink
    • https://drive.google.com/uc?export=download&id=###

File with several URLs

If you take a file including URLs, you lot can input the URL data using standard input and pipe as follows. If wrong URL is included, the URL is skipped.

$ true cat sample.txt              |              goodls

or

sample.txt

              https://docs.google.com/spreadsheets/d/#####/edit?usp=sharing https://docs.google.com/document/d/#####/edit?usp=sharing https://docs.google.com/presentation/d/#####/edit?usp=sharing                          

When y'all download shared files from Google Drive, please confirm whether the files are shared.

two. Download all files from shared folder

When above structure is downloaded, the command is similar beneath. At that fourth dimension, the folder ID is the folder ID of "sampleFolder1".

Files are downloaded from the shared folder. In this demonstration, the false folder ID and API fundamental are used.

Think API key

In order to use this, please recollect API key as the following flow.

1. Login to Google.

2. Access to https://panel.cloud.google.com/?hl=en.

3. Click select project at the right side of "Google Cloud Platform" of upper left of window.

   • If you cannot see the project, please try to access to https://console.cloud.google.com/cloud-resource-manager?hl=en. You can also create new project at there. - When y'all created new project there, delight click the left of "Google Cloud Platform". You can encounter it like 3 horizontal lines. By this, a side bar is opened. At there, delight select "API & Services" -> "Library". Subsequently this, please check the post-obit flow from Click "Library" at left side..

4. Click "NEW PROJECT"

   1. Input "Project Name".
   2. Click "CREATE".
   3. Open up the created project.
   4. Click "Enable APIs and become credentials like keys".
   5. Click "Library" at left side.
   6. Input "Bulldoze API" in "Search for APIs & Services".
   7. Click "Google Bulldoze API".
   8. Click "ENABLE".
   9. Back to https://console.cloud.google.com/?hl=en.
   10. Click "Enable APIs and go credentials like keys".
   11. Click "Credentials" at left side.
   12. Click "Create credentials" and select API key.
   13. Re-create the API key. You can use this API central.

Registering API fundamental to environs variable

• When API central is registered to the environs variable. When GOODLS_APIKEY equally the environment variable is set API key, goodls uses API primal from the environment variable.
• If API key is used with the pick at the command line, the priority of pick is college than the environs variable.

For example, in the case of fustigate, delight add a following code to .bashrc.

              export              GOODLS_APIKEY=                ### your API key ###            

Download

When the URL of shared binder is https://bulldoze.google.com/drive/folders/#####?usp=sharing, you can download all files in the folder by the following command.

$ goodls -u https://drive.google.com/drive/folders/                #####?usp=sharing -key [APIkey]            

• Project files cannot exist downloaded past API key. If y'all desire to download the project files, yous tin can download them by ggsrun, because ggsrun uses OAuth2.
• This new function uses the Go library of go-getfilelist.
• When the option of --NoProgres, -np is used, the progress information is non seen. This is a silent mode.
• If the files which are tried to be downloaded are existing, an error occurs. But when you lot employ the option --overwrite and --skip, the files are overwritten and skipped, respectively.
• If you utilise the selection -m, y'all tin retrieve only files with the specific mimeType from the folder. For example, when you want to download PDF and PNG files, please run similar beneath.
  • $ goodls -u [URL] -fundamental [APIkey] -m "awarding/pdf,image/png"

Call up information of file and binder

When you want to retrieve the information of file and folder, you can do it every bit follows.

For file

$ goodls -u https://docs.google.com/spreadsheets/d/                #####/edit?usp=sharing -key [APIkey] -i            

For folder

$ goodls -u https://drive.google.com/bulldoze/folders/                #####?usp=sharing -key [APIkey] -i            

3. Resumable download of shared file

When you use this option, at starting time, please retrieve API primal. About how to retrieve API key, you can come across at hither.

When yous desire to download 100 MBytes of the shared file, you can use the following command.

$ goodls -u [URL of shared file on Google Drive] -fundamental [APIkey] -r 100m

• Please use the selection -r. In this sample, 100m ways to download 100 MBytes of the shared file.
  • If you desire to download one GB, please employ -r 1g.
  • If yous use -r 1000000, 1 MByte of the file will exist able to be downloaded.

You lot tin can see the actual running of this option at the following demonstration film.

In this demonstration, the following command is run 3 times.

$ goodls -u https://drive.google.com/drive/folders/abcdefg?usp=sharing -key htjklmn -r 80m

• At 1st run, the information of 0 - 80 MBytes is downloaded.
  • Yous can see New download at "Current status".
• At 2d run, the information of lxxx - last is downloaded.
  • Yous can see Resumable download at "Current status".
• At tertiary run, the download has already been done. And so the checksum is shown.
  • You can see Download has already done. at "Electric current status".

Annotation

• Reason that API key is used for this.
  • When it accesses to the shared file without the dominance, the file size and md5checksum cannot be retrieved. And so in order to use Drive API, I adopted to use API key.
• Reason that the download size is inputted every time.
  • When this pick is run 1 time, one quota is used for Drive API. Then I adopted this way.

Q&A

• I want to download shared projects from user'southward Google Drive.

  • You tin can download shared projects using ggsrun.
  • ggsrun can also download shared files from other user'southward Google Drive using Drive API which needs the access token.

• I desire to download all files including the standalone projects from the shared binder and own folder.

  • You can achieve it using ggsrun.

• I want to use this with "Dockerfile build".

  • In this example, information technology has already been mentioned by rahulpowar at #15. Delight check information technology.

---

Licence

MIT

Writer

Tanaike

If y'all accept whatsoever questions and commissions for me, experience gratis to tell me.

Update History

• v1.0.0 (January ten, 2018)

  1. Initial release.

• v1.0.one (January 11, 2018)

  1. In order to download several files, a datafile including URLs using Standard Input and Pipe accept gotten to be able to be inputted.

• v1.0.2 (May x, 2018)

  1. Files with big size has gotten to exist able to be used.
     • In lodge to download files with big size (several gigabytes), files are saved by chunks.

• v1.0.three (September 4, 2018)

  1. When the files are downloaded, the progress of downloading got to be able to exist displayed.
     • This demonstration can be seen at Demo.
     • If the new selection of --np is used, the progress is not displayed.

• v1.1.0 (November 4, 2018)

  1. Past using API key, files from the shared folder got to be able to be downloaded while keeping the folder structure.
     • This demonstration can exist seen at Demo.
  2. By using API central, the information of shared file and folder can be likewise retrieved.
  3. About the selection of --extension and -e, when -e ms is used, Google Docs (Certificate, Spreadsheet, Slides) are converted to Microsoft Docs (Word, Excel, Powerpoint), respectively.

• v1.1.1 (November 13, 2018)

                                        1. Version of [get-getfilelist](https://github.com/tanaikech/go-getfilelist) was updated. Considering the construction of ``bulldoze.File`` got to be able to be used, I also updated this awarding.                                  

• v1.2.0 (November 24, 2018)

  1. By using API key, the shared large files tin can be run the resumable download.
     • This demonstration can exist seen at Demo.

• v1.2.1 (Nov 25, 2018)

  1. API key got to exist able to exist used past an environment variable. When GOODLS_APIKEY equally the environment variable is prepare API key, goodls uses API fundamental from the surroundings variable.

• v1.2.ii (December 12, 2018)

  1. When files are downloaded from a specific folder, it got to be able to select mimeType. By this, files with the specific mimeType in the specific folder can be retrieved. For this, I updated the go library go-getfilelist.
     • $ goodls -u [URL] -key [APIkey] -thousand "application/pdf,image/png"

• v1.2.3 (October 31, 2019)

  1. An pick for inputting the directory to salve the downloaded files was added. When this option is not used, the files are saved to the current working directory.
     • $ goodls -u [URL] -d [directory]

• v1.2.four (January 3, 2020)

  1. It seems that the specification of github.com/urfave/cli was changed by the update of https://github.com/urfave/cli. By this, when become become -u github.com/tanaikech/goodls is run, an error occurred. So this error was removed.

• v1.2.v (January 29, 2020)

  1. An option for selecting whether the superlative directory is created was added.
     • $ goodls -u [URL] --notcreatetopdirectory or $ goodls -u [URL] -ntd
     • When this option is NOT used (default situation), when a folder including sub-folders is downloaded, the superlative binder which is downloaded is created every bit the meridian directory under the working directory. When this option is used, the top directory is not created and all files and sub-folders nether the meridian folder are downloaded under the working directory.
     • This feature request was implemented.

• v1.2.vi (February 23, 2020)

  1. When the files are downloaded from the shared binder, if an error occurs, the download was stopped. In this thread, a request for skipping this error was proposed. This request was implemented at v1.2.6. Delight endeavour the option of --skiperror similar beneath.
     • $ goodls -u https://drive.google.com/drive/folders/###?usp=sharing --skiperror

• v1.2.7 (August 21, 2020)

  1. As the URL for downloading the files, webContentLink was added. So from this version, the URL of https://bulldoze.google.com/uc?consign=download&id=### got to be able to be used. This is the asking from this thread.

• v1.2.viii (February 17, 2022)

  1. Recently, it seems that the specification the process for downloading the shared file on Google Drive has been changed. Then I updated goodls for reflecting this. The usage of goodls is not changed.

• v2.0.0 (Feb 25, 2022)

  1. By changing the specification of methods, drive.New() and transport.APIKey were deprecated. By this, I updated get-getfilelist. In this version, I used this updated library to goodls. And also, bulldoze.NewService() is used instead of drive.New().

• v2.0.one (February 26, 2022)

  1. A issues for the resumable download was removed.

Meridian

hortfrournight61.blogspot.com

Source: https://github.com/tanaikech/goodls

Share this post