Include File Content in Network Logs

[TheTechromancer]

Hey, first off thanks for making this amazing tool!

I recently integrated Gowitness as a BBOT module, and am making heavy use of the feature that saves the DOM of each session to the sqlite database.

Similar to how the urls table contains a dom column, I would love it if the network_logs table also contained a field for the content of each of the files retrieved (JS, CSS, etc.)

GPT 4 has the following to say about it:

To capture the contents of network requests, we would need to use the network.Enable, network.GetResponseBody, and network.RequestWillBeSent functions from the network package.

Here's a general idea of how you might achieve this:

1. Enable network tracking. To start recording network events, we need to enable the network domain using network.Enable().

2. Attach an event handler. We can use the chromedp.ListenTarget function to attach a handler for network events. This handler will be called for each network event.

3. Handle RequestWillBeSent events. The network.RequestWillBeSent event is triggered when a network request is about to be sent. We'll store the request ids for later.

4. Handle LoadingFinished events. The network.LoadingFinished event is triggered when a network request has been fully loaded. When this event is triggered, we can get the response body for the request using the network.GetResponseBody function.

Here's an example of how you might implement these steps in your buildTasks function:

var requestIDs = make(map[network.RequestID]struct{})

actions := append(actions, chromedp.ActionFunc(func(ctx context.Context) error {
    // Enable network tracking
    err := network.Enable().Do(ctx)
    if err != nil {
        return err
    }

    // Attach an event handler
    chromedp.ListenTarget(ctx, func(ev interface{}) {
        switch ev := ev.(type) {
        case *network.EventRequestWillBeSent:
            // Store the request id when a request is about to be sent
            requestIDs[ev.RequestID] = struct{}{}
        case *network.EventLoadingFinished:
            // When a request has finished loading, get its response body
            if _, ok := requestIDs[ev.RequestID]; ok {
                go func() {
                    // GetResponseBody might block, so run it in a separate goroutine
                    body, err := network.GetResponseBody(ev.RequestID).Do(ctx)
                    if err != nil {
                        log.Printf("Could not get response body for request id %s: %v", ev.RequestID, err)
                    } else {
                        // Save the response body
                        // You might want to save it to a file or a database depending on your needs
                        fmt.Printf("Response body for request id %s: %s", ev.RequestID, body)
                    }
                }()
            }
        }
    })

    return nil
}))

// Continue with the rest of your actions...

Please note that this example prints out the response body to the console. You would need to replace that part with your own code to save the response body to a SQLite database or any other storage system you'd like to use. Also, please make sure to handle errors appropriately according to your needs.

Let me know if you'd be interested in having this as a feature, and I can work on tackling it. Thanks again!

[leonjza]

Added for v3.