opensource.google.com

Menu

Docs

Go

This page documents rules for adding third-party Go code to //piper/third_party/golang.

IMPORTANT: Read go/thirdparty first. When creating a new directory, set *********************************** in the CL. For Go-specific questions, email emailremoved@.

What it’s for

The third_party/golang directory is for code that is developed outside of Piper and imported into Piper using the //piper/third_party/golang/update.go tool. The purpose of this tree is to enforce common standards about how new versions of Go code are imported into Piper.

Open source code that is developed with Piper as the source of truth (and possibly exported) does not need to adhere to the requirements of the third_party/golang tree and should generally live elsewhere in third_party.

Code which is being temporarily developed in Piper with the intention of an eventual move to an external repository may be added to third_party/golang.

Go requirements

Go code checked in to third_party/golang must compile using the current google3 Go release (using blaze).

To add a new project, you must use //piper/third_party/golang/update.go:

  1. Add an entry in the goPackage map which is named appropriately.
  2. Create any needed filters (e.g. exclude vendor) and patches.
  3. Run update.go: go run third_party/golang/update.go --dir=$YOUR_NEW_DIR
  4. Create the other required files.
  5. Build and test (requires the BUILD file to have been added).

If update.go is problematic, you’ll either have to fix or modify update.go, ask emailremoved@ to change it for you, or request an exception to forego using update.go. (It is very unlikely that we will permit you to forego using update.go.)

Package naming

The update.go directory name should not start with “go”. If upstream is “gofoo”, just call it “foo” if foo is unique enough. If there are many foos, you can import “github.com/bar/go-foo” as “barfoo”. Groups of closely related packages are sometimes imported as “bar/foo”, though this is not typical.

Required files

Notes on specific files required by the Google-wide third_party policy:

  • OWNERS: create this yourself. The update.go tool won’t, but also won’t touch it.
  • BUILD file’s licenses section: create this yourself. You can do it after update.go creates the BUILD file.
  • METADATA file’s description: fill this in yourself. Also fill in the “security” section if required.

Making changes

After running update.go, you must not make any manual changes (except to BUILD files, in the case of glaze getting confused by package main files in a go_binary making too many go_binary rules). But running update.go repeatedly should not make any changes to your CL. Any changes that update.go makes to the upstream sources are considered canonical, and exempt from the usual go/pristinecopy policy to check in unmodified copies of the upstream source first. When sending out a CL importing a new project, explicitly state in the CL description that you tried running update.go again to ensure there were no further changes.

Filters

If you need to exclude certain files or folders from repository, use a filters file in a “patches” directory. See similar cases.

You will likely want to exclude files such as:

  • The vendor directory (dependencies should be added explicitly to third_party instead of vendored)
  • BUILD files (if the project uses bazel)
  • GitHub OWNERS files

If you modify the filters for a dir and re-run update.go, filtered out files will not be automatically deleted, you must delete them manually.

Syntax

The filters file a simple file with the following syntax:

  • Comments are any lines that matches ^\s*#
  • Exclude rules are any lines that matches ^exclude\s(.*) where the capture group is a regular expression used to exclude certain paths.
  • Include rules are any lines that matches ^include\s(.*) where the capture group is a regular expression used to include certain paths.

The filter is applied by evaluating every path in a repository the following way:

  • Paths are always rooted at the top of the repository with a leading ‘/’.
  • Paths end with a trailing ‘/’ if and only if referencing a directory.
  • The series of rules are evaluated in the order defined in the filters file until the first match is found, resulting in either an exclusion or inclusion.
  • There is an implicit “include all” rule at the bottom of every filter chain.

Patches

If you need to patch something, use a patch file in a “patches” directory. See similar cases. The update.go tool first pulls upstream, and then applies local patches. After adding a patch, it’s often a good idea to run update.go again to ensure that the patch applies cleanly and that tests still pass. Do not add patches for BUILD files. Try not to add patches at all if you can help it. It’s preferred to get your modification upstream first, and then pull it into google3.

Patches are applied by the patch command, so they must be in a format understood by patch. You can use git diff to generate these patches for example, but you must use --no-prefix, e.g.:

git diff --no-prefix > mypatch.patch

If you modify/add patches, rerunning update.go will apply the patches, i.e. you do not need to manually delete the affected files for them to be updated.

Testing

One note on blaze test vs “go test“: blaze test runs tests with a different working directory. “go test” always sets the working directory to the directory of the test. To work around this and let Go tests still find e.g. their “testdata” directories, either:

  • Add an import load("//third_party/golang:third_party_go_test.bzl", "third_party_go_test") to your BUILD file and use a third_party_go_test target (with parameter glaze_kind = "go_test") instead of go_test.
  • Add a file “google_init_test.go” to directories that need it (see examples). If a file begins with “google_” then update.go ignores it (doesn’t delete it).

Getting a review

To declare that you know the rules on this page (or at least to get a faster review), make your CL description like:

Import github.com/bar/foo as third_party/golang/foo

No manual changes. Running update.go --dir=foo doesn’t modify anything.

Checked that blaze {build,test} third_party/golang/foo/... works.

Troubleshooting

This section describes issues you may run into and how to resolve them.

go get failed

The project must be go-gettable via go get $YOUR_REPO/.... This recursive go get may cause problems with some repos. These issues must be fixed upstream and cannot be resolved with patches because the patches are applied after the code is fetched via go get.

If you run into this issue, you must submit a fix upstream before you can import the project via update.go

Except as otherwise noted, the content of this page is licensed under CC-BY-4.0 license. Third-party product names and logos may be the trademarks of their respective owners.