[Bug][CircleCI] Need better error handling in convertWorkflows when API is malformed

[magichair]

Search before asking

• I had searched in the issues and found no similar issues.

What happened

Running off of devlake.docker.scarf.sh/apache/devlake:main_250221_154524_8f654df I am unable to do an initial Collect Data of a CircleCI project (without some serious manual DB gymnastics during the sync).

Logs from the convertWorkflows job failure:

devlake-1  | time="2025-02-24 21:46:16" level=error msg=" [pipeline service] [pipeline #10] [task #74] run task failed with panic\n\tcaused by: run task failed with panic (github.com/apache/incubator-devlake/core/models/common.(*Iso8601Time).ToTime:120)\n\tWraps: (2) runtime error: invalid memory address or nil pointer dereference\n\tError types: (1) *hintdetail.withDetail (2) runtime.errorString"

For some reason the CreatedDate comes back as Null (well, a lot of things are missing in this particular entry).

Here's what I've found so far digging into this particular entry in the raw and tool tables and also from the CircleCI API directly for this weird workflow.

Here's what CircleCI tells me about this workflow from https://circleci.com/api/v2/pipeline/<GUID REDACTED>/workflow (fields redacted):

{
  "next_page_token" : null,
  "items" : [ {
    "pipeline_id" : "<GUID REDACTED>",
    "id" : null,
    "name" : null,
    "project_slug" : "gh/org-name/repo-name",
    "status" : null,
    "started_by" : null,
    "pipeline_number" : 6561,
    "created_at" : null,
    "stopped_at" : null
  } ]
}

What do you expect to happen

I expect an initial sync of CircleCI to complete successfully.

How to reproduce

Unfortunately, I don't have a public project with pipelines failing. However, there may be a way to write a test case using the raw CircleCI API response provided above (with a mock GUID).

Anything else

No response

Version

devlake.docker.scarf.sh/apache/devlake:main_250221_154524_8f654df

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[dosubot]

Hey there, @magichair! I'm a bot here to help you with bugs, questions, and becoming a contributor. Let's tackle this issue together while we wait for a human maintainer.

To improve error handling in the convertWorkflows function when the CircleCI API response is malformed, you can add a check for null values, particularly for fields like CreatedDate. Here's an example of how you can modify the Convert function to handle null CreatedDate:

Convert: func(inputRow interface{}) ([]interface{}, errors.Error) {
    userTool := inputRow.(*models.CircleciWorkflow)
    
    // Check if CreatedDate is null
    if userTool.CreatedDate == nil {
        return nil, errors.New("CreatedDate is null in the CircleCI API response")
    }
    
    createdAt := userTool.CreatedDate.ToTime()
    pipeline := &devops.CICDPipeline{
        DomainEntity: domainlayer.DomainEntity{
            Id: getWorkflowIdGen().Generate(data.Options.ConnectionId, userTool.Id),
        },
        Name:        userTool.Name,
        DurationSec: userTool.DurationSec,
        TaskDatesInfo: devops.TaskDatesInfo{
            CreatedDate:  createdAt,
            StartedDate:  &createdAt,
            FinishedDate: userTool.StoppedDate.ToNullableTime(),
        },
        CicdScopeId: getProjectIdGen().Generate(data.Options.ConnectionId, userTool.ProjectSlug),
        Status: devops.GetStatus(&devops.StatusRule{
            Done:    []string{"canceled", "failed", "failing", "success", "not_run", "error"},
            Default: devops.STATUS_OTHER,
        }, userTool.Status),
        OriginalStatus: userTool.Status,
        Result: devops.GetResult(&devops.ResultRule{
            Success: []string{"success"},
            Failure: []string{"failed", "failing", "error"},
            Default: devops.RESULT_DEFAULT,
        }, userTool.Status),
        DisplayTitle: fmt.Sprintf("%s#%d", userTool.Name, userTool.PipelineNumber),
    }
    
    if data.Options.ScopeConfig.DeploymentPattern != nil || data.Options.ScopeConfig.ProductionPattern != nil {
        pipeline.Type = data.RegexEnricher.ReturnNameIfMatched(devops.DEPLOYMENT, userTool.Name)
        pipeline.Environment = data.RegexEnricher.ReturnNameIfOmittedOrMatched(devops.PRODUCTION, userTool.Name)
    }
    
    result := make([]interface{}, 0, 2)
    result = append(result, pipeline)

    if p, err := findPipelineById(db, userTool.PipelineId); err == nil {
        if p.Vcs.Revision != "" {
            result = append(result, &devops.CiCDPipelineCommit{
                PipelineId:   pipeline.Id,
                CommitSha:    p.Vcs.Revision,
                Branch:       p.Vcs.Branch,
                RepoId:       p.Vcs.OriginRepositoryUrl,
                RepoUrl:      p.Vcs.OriginRepositoryUrl,
                DisplayTitle: pipeline.DisplayTitle,
            })
        }
    }
    return result, nil
},

This code snippet adds a check for CreatedDate being null and returns an error if it is, ensuring that the function handles malformed API responses gracefully ^[1].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}