Advanced Tools and Troubleshooting

NetBox Enterprise is designed to harness the power of Kubernetes while minimizing the amount of work the average person needs to manage it. However, sometimes it's still useful or necessary to peek under the hood.

Applications

The following applications are used for various facets of administration:

• kubectl*: CLI for interacting with clusters.
• preflight*: CLI for manually running preflight validation checks. Install by running:
  curl https://krew.sh/preflight | bash
• support-bundle*: CLI for manually generating support bundles. Install by running:
  curl https://krew.sh/support-bundle | bash
• k9s: a TUI for managing and viewing cluster resources.

* provided by the NetBox Enterprise install

Accessing Your Cluster from the Command Line

If you are running the NetBox Enterprise installer, you will need to first execute a command to get a shell environment that knows how to interact with it. To do this, run:

/var/lib/embedded-cluster/bin/netbox-enterprise shell

Accessing Your Cluster from the Web

The admin console is always available at https://your-cluster-host-or-ip:30000/ in a default NetBox Enterprise install.

Generating a Support Bundle

Command-Line

Access your cluster from the shell. If you haven't installed the support-bundle package, do so now.

Run:

kubectl support-bundle /var/lib/embedded-cluster/support/host-support-bundle.yaml --load-cluster-specs

This will run a series of tests and provide a short summary of what it found, much like the preflights that run during install and when deploying. When complete, the TUI will display, like so:

Typing s to save will save a short summary .txt file of the output you can see in the TUI. Typing q to quit will quit without saving the summary. In both cases, a .tar.gz support bundle will also be created in the current directory. This file can be used by NetBox Labs to troubleshoot your system.

Debug Logging

⚠️ Warning: Do not enable debug logging in production environments. Debug logging can expose sensitive information to unauthenticated users and significantly impact system performance.

Enable debug logging to increase log verbosity for troubleshooting issues. Debug logs provide detailed information about system operations and can help diagnose problems.

Enabling Debug Mode

1. Access the admin console
2. Navigate to "Config" section
3. Find "Enable Debug Logging" toggle
4. Enable the toggle
5. Click "Save Config"
6. Deploy the updated configuration

Viewing Debug Logs

Access the logs using:

kubectl logs netbox-enterprise-<pod_id> -n kotsadm

Log example:

Adjusting Worker Pod Resource Limits

In certain scenarios, such as syncing large Git repositories or processing resource-intensive operations, you may need to increase the resource limits allocated to worker pods. The default resource configuration may be insufficient for these specialized workloads.

When to Adjust Worker Limits

You should consider adjusting worker pod limits when experiencing:

• Git synchronization timeouts: Large repositories (>1GB) failing to sync
• Out-of-memory (OOM) errors: Worker pods being killed during operations
• Slow background job processing: Jobs taking excessive time to complete
• Large file handling: Processing repositories with files >50MB

Adjusting Resources via Admin Console

Configure worker pod resources through the admin console:

1. Access the admin console at https://your-cluster-host-or-ip:30000/
2. Navigate to Config section
3. Under Resources, select an appropriate resource preset:
   • Regular (up to 1.5Gi mem per replica): Default configuration for standard workloads
   • Large (up to 3Gi mem per replica): Moderate to heavy workloads
   • Extra Large (up to 6Gi mem per replica): Very large repositories and intensive operations
   • 2x Large (up to 12Gi mem per replica): Maximum performance for extremely large repositories (>1GB)
4. Click Save Config
5. Deploy the updated configuration

2x Large resource allocation (recommended for large repositories >1GB):

Worker Resources:
  limits:
    memory: "12Gi" (12Gi per replica)

Verifying Configuration

After deploying the new resource configuration, verify the changes:

# Check current worker resource limits
kubectl -n kotsadm get deployment netbox-enterprise-worker -o yaml \
  | yq -r '.spec.template.spec.containers[] | select(.name == "netbox-worker") | .resources'

# Monitor worker pod status
kubectl -n kotsadm get pods -l app=netbox-enterprise-worker

# Watch for successful pod restart with new limits
kubectl -n kotsadm get pods -w

Testing Large Repository Sync

After increasing resource allocation, test with your large repository:

1. Navigate to Operations → Integrations → Data Sources in NetBox
2. Add your Git repository as a data source
3. Click Sync and monitor the operation
4. Verify successful completion in the NetBox UI or worker logs

Expected behavior with 2x Large resources:

• Repositories up to 2GB should sync successfully
• Files up to 50MB+ can be processed
• No OOM (out of memory) kills during sync operations

Additional Troubleshooting

If sync continues to fail even with 2x Large resource allocation:

• Exclude unnecessary files: Use .gitignore patterns in your repository to exclude very large binary files that aren't needed for NetBox
• Split repositories: Consider dividing very large monorepos into smaller, more focused repositories
• Contact support: Reach out to NetBox Labs support for specialized configuration assistance with extremely large datasets

Tip: Most large repository sync issues are resolved by selecting Extra Large or 2x Large resource presets in the admin console.