Changes planned when no actual changes made

[mlcooper]

Terraform Version

Terraform v1.3.7
on windows_amd64
+ provider registry.terraform.io/hashicorp/archive v2.3.0
+ provider registry.terraform.io/hashicorp/aws v4.58.0

Terraform Configuration Files

data "aws_iam_policy_document" "cloudwatch_to_fh_access_policy" {
  statement {
    actions = [
      "firehose:*",
    ]

    effect = "Allow"

    resources = [
      aws_kinesis_firehose_delivery_stream.kinesis_firehose.arn,
    ]
  }

  statement {
    actions = [
      "iam:PassRole",
    ]

    effect = "Allow"

    resources = [
      aws_iam_role.cloudwatch_to_firehose_trust.arn,
    ]
  }
}

resource "aws_iam_role" "kinesis_firehose" {
  name        = var.kinesis_firehose_role_name
  description = "IAM Role for Kenisis Firehose"

  assume_role_policy = <<POLICY
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Principal": {
        "Service": "firehose.amazonaws.com"
      },
      "Action": "sts:AssumeRole",
      "Effect": "Allow"
    }
  ]
}
POLICY

  tags = var.tags
}

data "aws_iam_policy_document" "kinesis_firehose_policy_document" {
  statement {
    actions = [
      "s3:AbortMultipartUpload",
      "s3:GetBucketLocation",
      "s3:GetObject",
      "s3:ListBucket",
      "s3:ListBucketMultipartUploads",
      "s3:PutObject",
    ]

    resources = [
      aws_s3_bucket.kinesis_firehose_s3_bucket.arn,
      "${aws_s3_bucket.kinesis_firehose_s3_bucket.arn}/*",
    ]

    effect = "Allow"
  }

  statement {
    actions = [
      "lambda:InvokeFunction",
      "lambda:GetFunctionConfiguration",
    ]

    resources = [
      "${aws_lambda_function.firehose_lambda_transform.arn}:$LATEST",
    ]
  }

  statement {
    actions = [
      "logs:PutLogEvents",
    ]

    resources = [
      aws_cloudwatch_log_group.kinesis_logs.arn,
      aws_cloudwatch_log_stream.kinesis_logs.arn,
    ]

    effect = "Allow"
  }
}

resource "aws_kinesis_firehose_delivery_stream" "kinesis_firehose" {
  name        = var.firehose_name
  destination = "splunk"

  s3_configuration {
    role_arn           = aws_iam_role.kinesis_firehose.arn
    prefix             = var.s3_prefix
    bucket_arn         = aws_s3_bucket.kinesis_firehose_s3_bucket.arn
    buffer_size        = var.kinesis_firehose_buffer
    buffer_interval    = var.kinesis_firehose_buffer_interval
    compression_format = var.s3_compression_format
  }

  dynamic "server_side_encryption" {
    for_each = var.firehose_server_side_encryption_enabled == true ? [1] : []
    content {
      enabled  = var.firehose_server_side_encryption_enabled
      key_type = var.firehose_server_side_encryption_key_type
      key_arn  = var.firehose_server_side_encryption_key_arn
    }
  }

  splunk_configuration {
    hec_endpoint               = var.hec_url
    hec_token                  = var.hec_token != null ? module.hec_token_kms_secret[0].hec_token_kms_secret : var.self_managed_hec_token
    hec_acknowledgment_timeout = var.hec_acknowledgment_timeout
    hec_endpoint_type          = var.hec_endpoint_type
    s3_backup_mode             = var.s3_backup_mode

    processing_configuration {
      enabled = "true"

      processors {
        type = "Lambda"

        parameters {
          parameter_name  = "LambdaArn"
          parameter_value = "${aws_lambda_function.firehose_lambda_transform.arn}:$LATEST"
        }
        parameters {
          parameter_name  = "RoleArn"
          parameter_value = aws_iam_role.kinesis_firehose.arn
        }
        dynamic "parameters" {
          for_each = var.lambda_processing_buffer_size_in_mb != null ? [1] : []
          content {
            parameter_name  = "BufferSizeInMBs"
            parameter_value = var.lambda_processing_buffer_size_in_mb
          }
        }
      }
    }

    cloudwatch_logging_options {
      enabled         = var.enable_fh_cloudwatch_logging
      log_group_name  = aws_cloudwatch_log_group.kinesis_logs.name
      log_stream_name = aws_cloudwatch_log_stream.kinesis_logs.name
    }
  }

  tags = var.tags
}

resource "aws_lambda_function" "firehose_lambda_transform" {
  function_name                  = var.lambda_function_name
  description                    = "Transform data from CloudWatch format to Splunk compatible format"
  filename                       = data.archive_file.lambda_function.output_path
  role                           = aws_iam_role.kinesis_firehose_lambda.arn
  handler                        = local.lambda_function_handler
  source_code_hash               = data.archive_file.lambda_function.output_base64sha256
  runtime                        = var.nodejs_runtime
  timeout                        = var.lambda_function_timeout
  reserved_concurrent_executions = var.lambda_reserved_concurrent_executions

  dynamic "tracing_config" {
    for_each = var.lambda_tracing_config == null ? [] : [1]
    content {
      mode = var.lambda_tracing_config
    }
  }

  tags = var.tags
}

# kinesis-firehose-cloudwatch-logs-processor.js was taken by copy/paste from the AWS UI.  It is predefined blueprint
# code supplied to AWS by Splunk.
data "archive_file" "lambda_function" {
  type        = "zip"
  source_file = local.lambda_function_source_file
  output_path = "${path.module}/files/kinesis-firehose-cloudwatch-logs-processor.zip"
}
...terraform config...

The rest of the config are resources not involved in the issue I'm having.

Debug Output

Please see Actual Behavior below.

Expected Behavior

ONLY module.kinesis-firehose-splunk.aws_kinesis_firehose_delivery_stream.kinesis_firehose and module.kinesis-firehose-splunk.aws_lambda_function.firehose_lambda_transform would show pending changes in the terraform plan.

Actual Behavior

I can understand how module.kinesis-firehose-splunk.aws_kinesis_firehose_delivery_stream.kinesis_firehose and module.kinesis-firehose-splunk.aws_lambda_function.firehose_lambda_transform are valid changes, but the other resources that are changing do NOT actually have any changes to make. They all say this:

(depends on a resource or a module with changes pending)

Output:

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  ~ update in-place
 <= read (data resources)

Terraform will perform the following actions:

  # module.kinesis-firehose-splunk.data.aws_iam_policy_document.cloudwatch_to_fh_access_policy will be read during apply
  # (depends on a resource or a module with changes pending)
 <= data "aws_iam_policy_document" "cloudwatch_to_fh_access_policy" {
      + id   = (known after apply)
      + json = (known after apply)

      + statement {
          + actions   = [
              + "firehose:*",
            ]
          + effect    = "Allow"
          + resources = [
              + "arn:aws:firehose:us-west-2:914025435615:deliverystream/kinesis-firehose-to-splunk",
            ]
        }
      + statement {
          + actions   = [
              + "iam:PassRole",
            ]
          + effect    = "Allow"
          + resources = [
              + "arn:aws:iam::914025435615:role/CloudWatchToSplunkFirehoseTrust",
            ]
        }
    }

  # module.kinesis-firehose-splunk.data.aws_iam_policy_document.kinesis_firehose_policy_document will be read during apply
  # (depends on a resource or a module with changes pending)
 <= data "aws_iam_policy_document" "kinesis_firehose_policy_document" {
      + id   = (known after apply)
      + json = (known after apply)

      + statement {
          + actions   = [
              + "s3:AbortMultipartUpload",
              + "s3:GetBucketLocation",
              + "s3:GetObject",
              + "s3:ListBucket",
              + "s3:ListBucketMultipartUploads",
              + "s3:PutObject",
            ]
          + effect    = "Allow"
          + resources = [
              + "arn:aws:s3:::coopm017-test-splunk-fh",
              + "arn:aws:s3:::coopm017-test-splunk-fh/*",
            ]
        }
      + statement {
          + actions   = [
              + "lambda:GetFunctionConfiguration",
              + "lambda:InvokeFunction",
            ]
          + resources = [
              + "arn:aws:lambda:us-west-2:914025435615:function:kinesis-firehose-transform:$LATEST",
            ]
        }
      + statement {
          + actions   = [
              + "logs:PutLogEvents",
            ]
          + effect    = "Allow"
          + resources = [
              + "arn:aws:logs:us-west-2:914025435615:log-group:/aws/kinesisfirehose/kinesis-firehose-to-splunk",
              + "arn:aws:logs:us-west-2:914025435615:log-group:/aws/kinesisfirehose/kinesis-firehose-to-splunk:log-stream:SplunkDelivery",
            ]
        }
    }

  # module.kinesis-firehose-splunk.data.aws_iam_policy_document.lambda_policy_doc will be read during apply
  # (depends on a resource or a module with changes pending)
 <= data "aws_iam_policy_document" "lambda_policy_doc" {
      + id   = (known after apply)
      + json = (known after apply)

      + statement {
          + actions   = [
              + "logs:GetLogEvents",
            ]
          + effect    = "Allow"
          + resources = [
              + "arn:aws:logs:us-west-2:914025435615:log-group:/aws/apigateway/welcome:*",
            ]
        }
      + statement {
          + actions   = [
              + "firehose:PutRecordBatch",
            ]
          + resources = [
              + "arn:aws:firehose:us-west-2:914025435615:deliverystream/kinesis-firehose-to-splunk",
            ]
        }
      + statement {
          + actions   = [
              + "logs:PutLogEvents",
            ]
          + effect    = "Allow"
          + resources = [
              + "*",
            ]
        }
      + statement {
          + actions   = [
              + "logs:CreateLogGroup",
            ]
          + effect    = "Allow"
          + resources = [
              + "*",
            ]
        }
      + statement {
          + actions   = [
              + "logs:CreateLogStream",
            ]
          + effect    = "Allow"
          + resources = [
              + "*",
            ]
        }
    }

  # module.kinesis-firehose-splunk.aws_iam_policy.cloudwatch_to_fh_access_policy will be updated in-place
  ~ resource "aws_iam_policy" "cloudwatch_to_fh_access_policy" {
        id          = "arn:aws:iam::914025435615:policy/KinesisCloudWatchToFirehosePolicy"
        name        = "KinesisCloudWatchToFirehosePolicy"
      ~ policy      = jsonencode(
            {
              - Statement = [
                  - {
                      - Action   = "firehose:*"
                      - Effect   = "Allow"
                      - Resource = "arn:aws:firehose:us-west-2:914025435615:deliverystream/kinesis-firehose-to-splunk"
                      - Sid      = ""
                    },
                  - {
                      - Action   = "iam:PassRole"
                      - Effect   = "Allow"
                      - Resource = "arn:aws:iam::914025435615:role/CloudWatchToSplunkFirehoseTrust"
                      - Sid      = ""
                    },
                ]
              - Version   = "2012-10-17"
            }
        ) -> (known after apply)
        tags        = {}
        # (5 unchanged attributes hidden)
    }

  # module.kinesis-firehose-splunk.aws_iam_policy.kinesis_firehose_iam_policy will be updated in-place
  ~ resource "aws_iam_policy" "kinesis_firehose_iam_policy" {
        id        = "arn:aws:iam::914025435615:policy/KinesisFirehose-Policy"
        name      = "KinesisFirehose-Policy"
      ~ policy    = jsonencode(
            {
              - Statement = [
                  - {
                      - Action   = [
                          - "s3:PutObject",
                          - "s3:ListBucketMultipartUploads",
                          - "s3:ListBucket",
                          - "s3:GetObject",
                          - "s3:GetBucketLocation",
                          - "s3:AbortMultipartUpload",
                        ]
                      - Effect   = "Allow"
                      - Resource = [
                          - "arn:aws:s3:::coopm017-test-splunk-fh/*",
                          - "arn:aws:s3:::coopm017-test-splunk-fh",
                        ]
                      - Sid      = ""
                    },
                  - {
                      - Action   = [
                          - "lambda:InvokeFunction",
                          - "lambda:GetFunctionConfiguration",
                        ]
                      - Effect   = "Allow"
                      - Resource = "arn:aws:lambda:us-west-2:914025435615:function:kinesis-firehose-transform:$LATEST"
                      - Sid      = ""
                    },
                  - {
                      - Action   = "logs:PutLogEvents"
                      - Effect   = "Allow"
                      - Resource = [
                          - "arn:aws:logs:us-west-2:914025435615:log-group:/aws/kinesisfirehose/kinesis-firehose-to-splunk:log-stream:SplunkDelivery",
                          - "arn:aws:logs:us-west-2:914025435615:log-group:/aws/kinesisfirehose/kinesis-firehose-to-splunk",
                        ]
                      - Sid      = ""
                    },
                ]
              - Version   = "2012-10-17"
            }
        ) -> (known after apply)
        tags      = {}
        # (4 unchanged attributes hidden)
    }

  # module.kinesis-firehose-splunk.aws_iam_policy.lambda_transform_policy will be updated in-place
  ~ resource "aws_iam_policy" "lambda_transform_policy" {
        id        = "arn:aws:iam::914025435615:policy/Kinesis-Firehose-to-Splunk-Policy"
        name      = "Kinesis-Firehose-to-Splunk-Policy"
      ~ policy    = jsonencode(
            {
              - Statement = [
                  - {
                      - Action   = "logs:GetLogEvents"
                      - Effect   = "Allow"
                      - Resource = "arn:aws:logs:us-west-2:914025435615:log-group:/aws/apigateway/welcome:*"
                      - Sid      = ""
                    },
                  - {
                      - Action   = "firehose:PutRecordBatch"
                      - Effect   = "Allow"
                      - Resource = "arn:aws:firehose:us-west-2:914025435615:deliverystream/kinesis-firehose-to-splunk"
                      - Sid      = ""
                    },
                  - {
                      - Action   = "logs:PutLogEvents"
                      - Effect   = "Allow"
                      - Resource = "*"
                      - Sid      = ""
                    },
                  - {
                      - Action   = "logs:CreateLogGroup"
                      - Effect   = "Allow"
                      - Resource = "*"
                      - Sid      = ""
                    },
                  - {
                      - Action   = "logs:CreateLogStream"
                      - Effect   = "Allow"
                      - Resource = "*"
                      - Sid      = ""
                    },
                ]
              - Version   = "2012-10-17"
            }
        ) -> (known after apply)
        tags      = {}
        # (4 unchanged attributes hidden)
    }

  # module.kinesis-firehose-splunk.aws_kinesis_firehose_delivery_stream.kinesis_firehose will be updated in-place
  ~ resource "aws_kinesis_firehose_delivery_stream" "kinesis_firehose" {
        id             = "arn:aws:firehose:us-west-2:914025435615:deliverystream/kinesis-firehose-to-splunk"
        name           = "kinesis-firehose-to-splunk"
        tags           = {}
        # (5 unchanged attributes hidden)

      ~ splunk_configuration {
          # Warning: this attribute value will no longer be marked as sensitive
          # after applying this change. The value is unchanged.
          ~ hec_token                  = (sensitive value)
            # (5 unchanged attributes hidden)

            # (2 unchanged blocks hidden)
        }

        # (2 unchanged blocks hidden)
    }

  # module.kinesis-firehose-splunk.aws_lambda_function.firehose_lambda_transform will be updated in-place
  ~ resource "aws_lambda_function" "firehose_lambda_transform" {
      ~ filename                       = ".terraform/modules/kinesis-firehose-splunk/files/kinesis-firehose-cloudwatch-logs-processor.zip" -> "../terraform-aws-kinesis-firehose-splunk/files/kinesis-firehose-cloudwatch-logs-processor.zip"
        id                             = "kinesis-firehose-transform"
      ~ last_modified                  = "2023-03-14T16:45:59.698+0000" -> (known after apply)
        tags                           = {}
        # (21 unchanged attributes hidden)

        # (2 unchanged blocks hidden)
    }

Plan: 0 to add, 5 to change, 0 to destroy.

Steps to Reproduce

terraform plan

Additional Context

No response

References

I am experiencing the same thing happening in hashicorp/terraform-provider-aws#29393, and it was suggested there to open an issue in this repo.

[jbardin]

Hi @mlcooper,

The additional plan notation of (depends on a resource or a module with changes pending) is trying to help explain that you structured the configuration such that the data sources depend on managed resources which have changes pending. If the data source depends on the result of a managed resource, it cannot be read until after that managed resource has been applied. If other managed resources depend on the result of the data source, they may not be able to fully determine what changes need to be made until after that data source can be read. In most cases those planned changes will turn into noop changes during apply, but it would be more optimal to not require the data sources to be deferred in the first place.

We use GitHub issues for tracking bugs and enhancements, rather than for questions. While we can sometimes help with certain simple problems here, it's better to use the community forum where there are more people ready to help.

Thanks!

[richardgavel-ordinaryexperts]

Here's where I could see a value to be added. It's known, clearly that the data document depends on specific resources that are changing as a result of the plan. Theoretically, it should be possible to infer "well, I'm dependent on a property which can be identified as something whose value is fixed upon creation (an ID or ARN), and won't change short of the resource being recreated (which the plan would be aware of, vs a change). Since the data document is ONLY dependent on fixed properties of resources that are changing, it could skip the re-evaluation of the data resource.

[jbardin]

@richardgavel-ordinaryexperts, if you only need to depend on the specific attributes changing, then terraform will already do what you suggest. The use of depends_on is for when there is no useful attribute to track, and you need to ensure the data source is still read after the changes are applied to account for any external side effects.

[jlm0x017]

@jbardin I'm confused by the explanation you're trying to give in your answer.

The requested feature suggests that the attributes of "ID"/"ARN" are fixed for all time. Therefore the data block should not have to change as it needs immutable attributes. The issue seems to demonstrate where the data block is not smart enough to see that those attributes do not change.

In your response, you seem to describe behaviour when attributes need to change. That's the opposite of what is desired. What's the expected behaviour for a datablock when it references a resource that is being updated in place. Does the terraform plan see, on an individual resource-attribute level, which attributes are changing and which are fixed?

The description of 'depends_on' behaviour seems tangential, as the code snippet above does not use that keyword.

[jbardin]

Thanks @jlm0x017, I think I did miss that interpretation of the original post. The output (depends on a resource or a module with changes pending) is attempting to show that the data sources cannot be known because something it depends on has pending changes triggered by depends_on. When all data sources are triggered like this, it is almost always because of an errant depends_on being used in a parent module call.

Reading more carefully here it looks like all data sources in question are directly referencing managed resources, which (unfortunately) implies the same thing as setting depends_on. See Data Resource Dependencies for more details on that.

If there is no use of depends_on in the parent modules, then collecting the references in local values should disassociate the data sources from the managed resources.

[mlcooper]

@jbardin I can confirm I did not have any depends_on statements in the parent module. The child module has one depends_on statement which is in this code example (depends_on = [aws_s3_bucket_versioning.versioning]).

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.