[bug] Sourcebot not indexing forks and archived repos.

[nvroot]

Bug

Sourcebot isn’t indexing forked or archived GitHub repositories, even though exclude.forks and exclude.archived are set to false. Total repos 100+ , indexed only 23 (own repos)

config.json:

{
  "$schema": "https://raw.githubusercontent.com/sourcebot-dev/sourcebot/main/schemas/v3/index.json",
  "connections": {
    "github-nvroot": {
      "type": "github",
      "token": {
        "env": "SOURCEBOT_GITHUB_TOKEN"
      },
      "users": [
        "nvroot"
      ],
      "exclude": {
        "archived": false,
        "forks": false,
        "size": {
          "max": 1073741824
        }
      }
    }
  },
  "models": [
    {
      "provider": "openrouter",
      "model": "openrouter-model",
      "displayName": "openrouter-model",
      "token": {
        "env": "OPENROUTER_TOKEN"
      },
      "baseUrl": "https://openrouter.ai/api/v1"
    }
  ],
  "settings": {
    "reindexIntervalMs": 86400000
  }
}

docker-compose:

services:
  sourcebot:
    image: 'ghcr.io/sourcebot-dev/sourcebot:latest'
    container_name: sourcebot
    restart: unless-stopped
    expose:
      - 3000
    volumes:
      - './data:/data'
    environment:
      - CONFIG_PATH=/data/config.json
      - DATA_DIR=/data
      - SOURCEBOT_TELEMETRY_DISABLED=true
      - SOURCEBOT_TENANCY_MODE=single
      - AUTH_URL=https://source.example.com
      - AUTH_SECRET=REDACTED_LENGTH_44 # openssl rand -base64 33
      - AUTH_CREDENTIALS_LOGIN_ENABLED=true
      - SOURCEBOT_GITHUB_TOKEN=REDACTED
      - OPENROUTER_TOKEN=REDACTED
    networks:
      - proxy
    healthcheck:
      test: ['CMD', 'curl', '-f', 'http://127.0.0.1:3000']
      interval: 60s
      timeout: 20s
      retries: 5
      start_period: 10s
networks:
  proxy:
    external: true

also tried:

        "archived": true,
        "forks": true,

To reproduce

1. Use the config above.
2. Set SOURCEBOT_GITHUB_TOKEN #Checked with Fine-Grained and Classic.

Sourcebot deployment information

Sourcebot version (Latest): 4.6.4

Additional information

Logs:

[Info] Sourcebot version: v4.6.4
[Info] Disabling telemetry since SOURCEBOT_TELEMETRY_DISABLED was set.
[Warning] SOURCEBOT_ENCRYPTION_KEY is not set.
[Info] Loading environment variables from /data/.sourcebot/.secret
localhost:5432 - no response
[Info] Waiting for the database to be ready...
2025-08-19 04:18:14.738 UTC [14] LOG:  starting PostgreSQL 16.9 on aarch64-alpine-linux-musl, compiled by gcc (Alpine 13.2.1_git20231014) 13.2.1 20231014, 64-bit
2025-08-19 04:18:14.739 UTC [14] LOG:  listening on IPv6 address "::1", port 5432
2025-08-19 04:18:14.739 UTC [14] LOG:  listening on IPv4 address "127.0.0.1", port 5432
2025-08-19 04:18:14.749 UTC [14] LOG:  listening on Unix socket "/run/postgresql/.s.PGSQL.5432"
2025-08-19 04:18:14.759 UTC [19] LOG:  database system was interrupted; last known up at 2025-08-19 04:06:38 UTC
2025-08-19 04:18:14.923 UTC [19] LOG:  database system was not properly shut down; automatic recovery in progress
2025-08-19 04:18:14.928 UTC [19] LOG:  redo starts at 0/19E1A70
2025-08-19 04:18:14.928 UTC [19] LOG:  invalid record length at 0/19E1B58: expected at least 24, got 0
2025-08-19 04:18:14.928 UTC [19] LOG:  redo done at 0/19E1B20 system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s
2025-08-19 04:18:14.937 UTC [17] LOG:  checkpoint starting: end-of-recovery immediate wait
2025-08-19 04:18:14.951 UTC [17] LOG:  checkpoint complete: wrote 3 buffers (0.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.004 s, sync=0.002 s, total=0.018 s; sync files=2, longest=0.001 s, average=0.001 s; distance=0 kB, estimate=0 kB; lsn=0/19E1B58, redo lsn=0/19E1B58
2025-08-19 04:18:14.958 UTC [14] LOG:  database system is ready to accept connections
localhost:5432 - accepting connections
Database 'sourcebot' already exists; skipping creation.
[Info] Running database migration...
Prisma schema loaded from prisma/schema.prisma
Datasource "db": PostgreSQL database "sourcebot", schema "public" at "localhost:5432"
2025-08-19 04:18:19.564 UTC [119] LOG:  could not receive data from client: Connection reset by peer

34 migrations found in prisma/migrations


No pending migrations to apply.
2025-08-19 04:18:20,165 CRIT Supervisor is running as root.  Privileges were not dropped because no user is specified in the config file.  If you intend to run as root, you can set user=root in the config file to avoid this message.
2025-08-19 04:18:20,169 INFO supervisord started with pid 7
2025-08-19 04:18:21,174 INFO spawned: 'backend' with pid 154
2025-08-19 04:18:21,183 INFO spawned: 'redis' with pid 155
2025-08-19 04:18:21,191 INFO spawned: 'web' with pid 158
2025-08-19 04:18:21,205 INFO spawned: 'zoekt' with pid 166
155:C 19 Aug 2025 04:18:21.208 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
155:C 19 Aug 2025 04:18:21.210 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
155:C 19 Aug 2025 04:18:21.210 * Redis version=7.2.9, bits=64, commit=00000000, modified=0, pid=155, just started
155:C 19 Aug 2025 04:18:21.210 * Configuration loaded
155:M 19 Aug 2025 04:18:21.211 * monotonic clock: POSIX clock_gettime
155:M 19 Aug 2025 04:18:21.212 * Running mode=standalone, port=6379.
155:M 19 Aug 2025 04:18:21.214 * Server initialized
155:M 19 Aug 2025 04:18:21.214 * Loading RDB produced by version 7.2.9
155:M 19 Aug 2025 04:18:21.214 * RDB age 8 seconds
155:M 19 Aug 2025 04:18:21.214 * RDB memory usage when created 1.73 Mb
155:M 19 Aug 2025 04:18:21.214 * Done loading RDB, keys loaded: 10, keys expired: 0.
155:M 19 Aug 2025 04:18:21.214 * DB loaded from disk: 0.000 seconds
155:M 19 Aug 2025 04:18:21.214 * Ready to accept connections tcp
[zoekt] | 2025/08/19 04:18:21 [INFO] loading 23 shard(s): 1_10_v16.00000.zoekt, 1_11_v16.00000.zoekt, 1_12_v16.00000.zoekt, 1_13_v16.00000.zoekt, 1_14_v16.00000.zoekt... 18 more
[web] |   ▲ Next.js 14.2.30
[web] |   - Local:        http://localhost:3000
[web] |   - Network:      http://0.0.0.0:3000
[web] | 
[web] |  ✓ Starting...
[web] |  ✓ Ready in 699ms
2025-08-19 04:18:22,412 INFO success: backend entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-08-19 04:18:22,413 INFO success: redis entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-08-19 04:18:22,413 INFO success: web entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-08-19 04:18:22,414 INFO success: zoekt entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
[web] | 2025-08-19T04:18:23.004Z info: [web-initialize] Upserted connection with name 'github-nvroot'. Connection ID: 1
[backend] | 2025-08-19T04:18:23.706Z info: [repo-manager] Validating indexed repos have shards...
[backend] | 2025-08-19T04:18:23.718Z info: [prometheus-client] Prometheus metrics server is running on port 3060
[backend] | 2025-08-19T04:18:23.723Z info: [backend-main] Connected to redis
[backend] | 2025-08-19T04:18:23.865Z info: [repo-manager] Found 23 repos in the DB marked as INDEXED
[backend] | 2025-08-19T04:18:23.867Z info: [repo-manager] Done validating indexed repos have shards
[backend] | 2025-08-19T04:18:24.737Z info: [connection-manager] Added job to queue for connection github-nvroot (id: 1)
[web] | [edge] Sentry was not initialized
[backend] | 2025-08-19T04:18:25.658Z info: [connection-manager] Deleted all RepoToConnection records for connection github-nvroot (id: 1) in 13.67885699999988ms
[backend] | 2025-08-19T04:18:25.677Z info: [connection-manager] Upserted repo nvroot/REDACTED (id: 1039013219) in 18.34402300000056ms
[backend] | 2025-08-19T04:18:25.691Z info: [connection-manager] Upserted repo nvroot/REDACTED (id: 946886522) in 13.537537000000157ms
[backend] | 2025-08-19T04:18:25.708Z info: [connection-manager] Upserted repo nvroot/REDACTED (id: 875824557) in 16.686421000000337ms
[backend] | 2025-08-19T04:18:25.725Z info: [connection-manager] Upserted repo nvroot/REDACTED (id: 1025947504) in 16.3785799999996ms
[backend] | 2025-08-19T04:18:25.740Z info: [connection-manager] Upserted repo nvroot/REDACTED (id: 830680405) in 14.530738000000383ms
[backend] | 2025-08-19T04:18:25.759Z info: [connection-manager] Upserted repo nvroot/REDACTED (id: 847327040) in 17.94534299999941ms
[backend] | 2025-08-19T04:18:25.775Z info: [connection-manager] Upserted repo nvroot/REDACTED (id: 846252950) in 14.132497999999941ms
[backend] | 2025-08-19T04:18:25.792Z info: [connection-manager] Upserted repo nvroot/REDACTED (id: 944649744) in 16.495339999999487ms
[backend] | 2025-08-19T04:18:25.804Z info: [connection-manager] Upserted repo nvroot/REDACTED (id: 842217756) in 11.702734000000419ms
[backend] | 2025-08-19T04:18:25.817Z info: [connection-manager] Upserted repo nvroot/REDACTED (id: 875827410) in 12.611935999999332ms
[backend] | 2025-08-19T04:18:25.837Z info: [connection-manager] Upserted repo nvroot/REDACTED (id: 847465708) in 19.419785000000047ms
[backend] | 2025-08-19T04:18:25.848Z info: [connection-manager] Upserted repo nvroot/REDACTED (id: 885205018) in 10.876373999999487ms
[backend] | 2025-08-19T04:18:25.865Z info: [connection-manager] Upserted repo nvroot/REDACTED (id: 902273709) in 15.555379000000357ms
[backend] | 2025-08-19T04:18:25.884Z info: [connection-manager] Upserted repo nvroot/REDACTED (id: 875832236) in 19.127583999999842ms
[backend] | 2025-08-19T04:18:25.898Z info: [connection-manager] Upserted repo nvroot/REDACTED (id: 883047689) in 12.978457000000162ms
[backend] | 2025-08-19T04:18:25.912Z info: [connection-manager] Upserted repo nvroot/REDACTED (id: 917454913) in 12.913335999999617ms
[backend] | 2025-08-19T04:18:25.926Z info: [connection-manager] Upserted repo nvroot/REDACTED (id: 878783606) in 14.412818000000698ms
[backend] | 2025-08-19T04:18:25.941Z info: [connection-manager] Upserted repo nvroot/REDACTED (id: 876569695) in 13.964898000000176ms
[backend] | 2025-08-19T04:18:25.964Z info: [connection-manager] Upserted repo nvroot/REDACTED (id: 875824643) in 22.731988999999885ms
[backend] | 2025-08-19T04:18:25.978Z info: [connection-manager] Upserted repo nvroot/REDACTED (id: 883086352) in 13.037337000000662ms
[backend] | 2025-08-19T04:18:25.991Z info: [connection-manager] Upserted repo nvroot/REDACTED (id: 929999586) in 11.463133999999627ms
[backend] | 2025-08-19T04:18:26.004Z info: [connection-manager] Upserted repo nvroot/REDACTED (id: 875824465) in 12.970495999999912ms
[backend] | 2025-08-19T04:18:26.016Z info: [connection-manager] Upserted repo nvroot/REDACTED (id: 1005764717) in 11.282693999999537ms
[backend] | 2025-08-19T04:18:26.016Z info: [connection-manager] Upserted 23 repos for connection github-nvroot (id: 1) in 357.17704800000047ms
[backend] | 2025-08-19T04:18:26.029Z info: [connection-manager] Connection sync job for connection github-nvroot (id: 1, jobId: 4) completed

[tchikh411]

Hi @brendan-kellam , I'll take this

[tchikh411]

@nvroot , can you confirm what permissions have you assigned to your PAT? I can access the archived repos with permissions:

• Read access to code and metadata

I am having trouble accessing the forked repos but it seems the problem is not with sourcebot but with the permissions. If I use the API directly (https://api.github.com/search/repositories?q=user:), I am not seeing forked repos which indicates the list of repo sourcebot is getting do not have forked repos. I am looking into it.