I’ve struggled in getting tracing setup in Go for Google’s App Engine Second Generation.
In the original app-engine (standard environment) all of my outgoing HTTP requests had really great tracing on them, letting me know where my time was going.
They could do this because my programs were forced to go through special libraries to perform any outgoing requests, and these libraries did tracing for me.
In the new second generation runtime I’m allowed to use normal libraries, as the sandboxing has been moved down a layer of abstraction. This is great, except now the tracing is up to me.
The docs directed me towards using OpenCensus to get tracing set up, but nothing I found seem to have every step in the same place (as far as I could tell).
I cobbled together instruction from documentation pages from either Google or OpenCensus and I have something working. Even better I feel like I understand it! The OpenCensus libraries I’m using are still in alpha, and the second-gen App Engine is still beta, so there is still time for the project’s documentation to all come together.
Somewhere in your init or main method you’ll want to register the Stackdriver exporter:
// import "contrib.go.opencensus.io/exporter/stackdriver"
// import "go.opencensus.io/trace"
exporter, err := stackdriver.NewExporter(stackdriver.Options{})
if err != nil {
log.Fatal(err)
}
trace.RegisterExporter(exporter)This tells the OpenCensus libraries how to export recorded traces to GCP.
OpenCensus supplies an implementation of http.Handler which will add tracing to incoming requests and (this is important) can be told how to extract any tracing-context from incoming HTTP request headers (this is the Propagation bit). The OpenCensus handler will then take the tracing-context and stuff it in to http.Request.Conext() to make it available to down-stream request-handlers.
This tracing-context is what links the traces you’re later going to export up to a specific incoming request to produce the nice visualization up above.
// import "contrib.go.opencensus.io/exporter/stackdriver/propagation"
// import "go.opencensus.io/plugin/ochttp"
func main() {
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintln(w, "Hello, world!")
})
handler := &ochttp.Handler{Propagation: &propagation.HTTPFormat{}}
port := "3000"
log.Fatal(http.ListenAndServe(fmt.Sprintf(":%s", port), handler))
}Above I’m directly serving requests using the OpenCensus handler, which by default redirects to http.DefaultServeMux. If you’re using a more complex handler you can add tracing as a middleware:
func main() {
handler := addTracing(/* your handler goes here */)
port := "3000"
log.Fatal(http.ListenAndServe(fmt.Sprintf(":%s", port), handler))
}
func addTracing(inner http.Handler) http.Handler {
if os.Getenv("GOOGLE_CLOUD_PROJECT") == "" {
// Only setup tracing if we think we're running on google cloud
return inner
}
return &ochttp.Handler{Handler: inner, Propagation: &propagation.HTTPFormat{}}
}I have tracing as the outer-most middleware because debugging my site is my current priority. You might want some basic authentication or verification checks prior to tracing so you don’t end up with spam in your traces, maybe? I’m not really sure what best practices are here, but it’s something to watch out for.
Logging traces to outgoing requests is done in two parts - modifying your http.Client to use the OpenCensus supplied http.RoundTripper, and then passing on the parent request Context in to any outgoing requests.
For example:
func demoHandler(w http.ResponseWriter, r *http.Request) {
req, err := http.NewRequest("GET", "http://example.com", nil)
if err != nil {
// Do something about the error
internalServerError(w, r, err)
return
}
// Wire up OpenCensus to your HTTP client
client := &http.Client{}
client.Transport = &ochttp.Transport{
Propagation: &propagation.HTTPFormat{},
}
// Pass in span-context from incoming
// request in to outgoing request
req = req.WithContext(r.Context())
resp, err := client.Do(req)
if err != nil {
// Do something about the error
internalServerError(w, r, err)
return
}
defer resp.Body.Close()
// Do something with response
}All of that to make pretty graphs show up in my GCP traces. As far as I can tell each of the above steps is required (except maybe setting up the Propagation field in ochttp.Transport - that’s only if you want to pass on your tracing-context to the destination of your HTTP request).