Prototype Log4j2 Appender by jack-berg · Pull Request #4375 · open-telemetry/opentelemetry-java-instrumentation

jack-berg · 2021-10-13T23:42:48Z

EDIT - keeping the description below around, but all the issues it mentions have been solved in the 1.9.0 release of the log SDK. This is no longer a prototype.

This is meant to prompt discussion. Proves out what an OpenTelemetryAppender might look like for log4j2. The idea behind this appender is that it delivers log records to a opentelemetry LogSink, where they can be batched and shipped out of process via an exporter, such as the recently added OTLP grpc and http log exporters.

One of the main problems I had to address was that log appenders are instantiated before the opentelemetry log sdk is available. To get around this, each OpenTelemetryAppender instance is statically registered after initialization. Once the otel log sdk is available, it is passed to each OpnTelemetryAppender instance via a static method, upon which log records start flowing.

Learnings so far:

LogRecord trace id and span id need to be nullable. And potentially other properties too. The otlp grpc and http exporters currently fail to serialize if span id and trace id are left as their default "" values. Solved here.
We need a InMemoryLoggingExporter for testing. Mocked it out in this code, and will contribute back to opentelemetry-java. Solved here.
We need a SimpleLogProcessor for testing. Mocked it out in this code, and will contribute back to opentelemetry-java. Solved here.
LogSinkSdkProvider.builder() is not public but needs to be for it to be useful. I'll make it so. Solved here.
LogSinkSdkProvider should be immutable after its created. Log processors should be registered with the builder, not with the instance. I'll make it so. Solved here.
Thought needs to be put into how resource and instrumentation library get attached to the log records. Right now and producer to LogSink is responsible for creating LogRecords with resource and instrumenation library attached. This means that components like the OpenTelemetryAppender need to maintain fields for these. Should LogSink accept LogRecords without resource and instrumenation library, and let the pipeline attach those?

jack-berg · 2021-10-13T23:46:45Z

+    def sdkSinkProvider = LogSinkSdkProvider.builder().build()
+    sdkSinkProvider.addLogProcessor(SimpleLogProcessor.create(logExporter))
+    def logSink = sdkSinkProvider.get(null, null)
+    OpenTelemetryLog4j.initialize(logSink, resource)


This is the best demonstration of the usage right now. When this is ready to open as a real PR, I'll add docs outlining usage to the readme.

A user would add the opentelemetry appender to their config:

<?xml version="1.0" encoding="UTF-8"?> <Configuration status="WARN" packages="io.opentelemetry.instrumentation.log4j.v2_13_2"> <Appenders> <Console name="Console" target="SYSTEM_OUT"> <PatternLayout pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} traceId: %X{trace_id} spanId: %X{span_id} flags: %X{trace_flags} - %msg%n" /> </Console> <OpenTelemetry name="OpenTelemetryAppender" /> </Appenders> <Loggers> <Root> <AppenderRef ref="OpenTelemetryAppender" leve="All" /> <AppenderRef ref="Console" level="All" /> </Root> </Loggers> </Configuration>

Then initialize the appenders by calling OpenTelemetryLog4j.initialize(LogSink, Resource) as early in the application lifecycle as possible.

Any logs recorded before initializing are dropped.

I think this is a reasonable approach for now. Is it possible to build more complex configuration that would allow log4j to run the initialization itself?

Yes that would be possible. Log4j appenders builders have annotation based scheme for configuration properties. However, if configuration for logging is complex like it is for metrics & tracing, it will be inconvenient to maintain. It would be almost like an autoconfiguration module just for log4j.

yeah, that makes sense. Maybe we provide a standard couple of options, but then fall back to the manual init if that doesn't work for people.

…nstrumentation into log4j2-otel-appender

jack-berg · 2021-11-12T21:31:50Z

Now that 1.9.0 has been released, I've updated this branch, refined the logic, and added more thorough tests.

Its now ready for review.

NOTE this still places an explicit dependency on the otel java 1.9.0 log SDK. I think the whole repo should be updated to use 1.9.0 before this is merged.

iNikem · 2021-11-13T08:03:51Z

 }

 dependencies {
+  api("io.opentelemetry:opentelemetry-sdk-logs:1.9.0-alpha")


That will make all clients of this library instrumentation depend on SDK. I assume all users must depend only on API.

Users need to call OpenTelemetryLog4j.initialize(SdkLogEmitterProvider) to wire opentelemetry into the appender so the log SDK is actually part of this module's API.

But this reminds me that I need to update the readme to show usage!

@iNikem Yeah - we have open-telemetry/opentelemetry-java#3807 to track fixing this.

Sleeping on this, I realized we can keep this code here by making this compileOnly - anyone adding the appender has to add the SDK to their classpath themselves.

Though I still think it might be simpler to continue to prototype with the code in the SDK repo. What do you think?

I'm definitely wary about adding an API API to support this. I think compileOnly and a reflective check for the presence of an SDK is a better way to go until we have a spec for a new logging API.

compileOnly dependency on the SDK sounds reasonable to me, and potentially a path forward for all appenders.

Though I still think it might be simpler to continue to prototype with the code in the SDK repo.

Prototyping in the SDK has the advantage that the SDK and the appender can develop in lockstep, rather than the appender waiting on SDK releases, which is nice. But I think a lot of the SDK flux has been flushed out (of course assuming no significant changes to the experimental log spec) and so I don't expect to see much value from having the appender in the SDK.

iNikem · 2021-11-13T08:05:56Z

+    if (level != null) {
+      builder.setSeverity(
+          LEVEL_SEVERITY_MAP.getOrDefault(level, Severity.UNDEFINED_SEVERITY_NUMBER));
+      builder.setSeverityText(logEvent.getLevel().name());


Suggested change

builder.setSeverityText(logEvent.getLevel().name());

builder.setSeverityText(level.name());

And shouldn't this text come from our severity and not logging framework level?

From the spec, the severity text field "is the original string representation of the severity as it is known at the source".

But this does bring up the point that I have made a number of subjective decisions in mapping between the Log4j2 LogEvent data model and the otel log data model. Please take a close look at the attribute name's I've chosen for various things and let me know if you agree / disagree.

mateuszrzeszutek · 2021-11-13T21:11:49Z

+  @GuardedBy("LOCK")
+  private static final List<OpenTelemetryAppender> APPENDERS = new ArrayList<>();
+
+  public static void initialize(SdkLogEmitterProvider sdkLogEmitterProvider) {


How is this going to work as a javaagent instrumentation? In the agent we keep all the SDK classes hidden in the agent classloader, they're basically unavailable from the application classloader. The agent won't be able to register its emitterProvider in the application's OpenTelemetryLog4j class.
Was the idea about "log appender api" completely scrapped? What happened with that?

We'll need it - open-telemetry/opentelemetry-java#3807

Took a stab at extracting out the appender API here.

anuraaga

I think we need a new artifact, maybe log4j-logssdk-prototype since we can't leak the SDK dependency into existing users of the log4j instrumentation.

If a temporary artifact in Maven Central is not palatable (it's not great TBH), I think we should hold off for the 1.9.0 release and prioritize open-telemetry/opentelemetry-java#3807. Sorry, didn't notice this ordering constraint until now.

We could still get a snapshot out sooner than later by

Adding log4j-logssdk-prototype after the instrumentation 1.9.0 release, and merging it in for 1.10, using the log appender API that must be ready by then
Adding a compileOnly dependency on log4j in the opentelemetry-sdk-logs artifact itself and put the code there.

The second option seems to allow the easiest iteration on snapshots.

anuraaga · 2021-11-15T02:16:16Z

 dependencies {
  api(project(":testing-common"))

+  api("io.opentelemetry:opentelemetry-sdk-logs:1.9.0-alpha")


Don't think we need version here since we apply BOM for io.opentelemetry. Similar for all usages

Yes, I'm just waiting for this project to update its BOM dependency on 1.9.0 and then will drop the version here.

anuraaga · 2021-11-15T02:17:19Z

 }

 dependencies {
+  api("io.opentelemetry:opentelemetry-sdk-logs:1.9.0-alpha")


@iNikem Yeah - we have open-telemetry/opentelemetry-java#3807 to track fixing this.

anuraaga · 2021-11-15T02:18:17Z

+final class LogEventMapper {
+
+  // Visible for testing
+  static final AttributeKey<String> ATTR_LOGGER_NAME = AttributeKey.stringKey("logger.name");


I don't see any semantic conventions for logs defined yet, can we leave them out until they're defined?

https://github.com/open-telemetry/opentelemetry-specification/tree/main/specification/logs/semantic_conventions

If leaving any, I guess logger.name, throwable.name, throwable.message are the only ones I'd keep as they may be necessary for playing with a prototype, but others seem too detailed, or unclear (e.g., I'm not familiar with term ndc)

ndc = Nested Diagnostic Context

Nested diagnostic context is thread context that acts like a stack. You push items onto it, and pop them off. Mapped diagnostic context is thread context that acts like a map. Frankly, I haven't used Nested Diagnostic Context, and have only used Mapped Diagnostic Context in a limited way in applications I've written.

I took the stance of making sure that all data in the log4j2 data model somehow was mapped into the otel log data model. You're suggesting the opposite, where only a limited part of the log4j2 data model is mapped. I see pros and cons of each stance:

If we only do a partial mapping of fields well defined in semantic conventions, then we're less likely to make breaking changes, though we can't eliminate breaking changes since there are no stable semantic conventions. We're also going to disappoint folks using parts of the log4j2 data model that are unmapped.

If we do a fully mapping, we basically guarantee breaking changes, but at least all the data gets mapped. And by at least making an attempt at mapping each field we can provide better feedback for the semantic conventions.

Personally, I prefer the full mapping approach published in an "alpha" artifact. Definitely interested in other perspectives on this though.

I've seen it's easier to add an attribute and not really go anywhere in the spec with it than the opposite. By not including too much, we can hear about what users really need and then build the spec around it, rather than being ad-hoc. While I don't like to disappoint folks, actually hearing about that disappointment is very valuable feedback.

That resonates with me. Also, if we were to do a complete mapping and the required semantic conventions never materialized, we may have to pull support for that mapping, which would be an unpleasant breaking change for users.

Updated to a minimalist mapping in new commit.

anuraaga · 2021-11-15T02:30:42Z

+                        SpanContext.create(traceId, spanId, traceFlags, TraceState.getDefault())));
+        builder.setContext(context);
+
+        contextMap.forEach(attributes::put);


I don't think we should fill arbitrary attributes, I don't think we ever do anything similar for spans

Kind of related to the the conversation about doing a complete mapping or not.

Not filling arbitrary attributes from mapped diagnostic context is limiting the usefulness of the log4j2 api. It means if you want to use otel logs in a more "machine generated events" type of way, that you'd want to use the otel log sdk in your application instead of the log4j2 api.

I think we'd generally use baggage for more arbitrary attributes? Though we haven't even specced it for spans. But I'd still suggest keeping things smaller for now to make sure the spec grows rather than going all in

anuraaga · 2021-11-15T02:36:02Z

+  @GuardedBy("LOCK")
+  private static final List<OpenTelemetryAppender> APPENDERS = new ArrayList<>();
+
+  public static void initialize(SdkLogEmitterProvider sdkLogEmitterProvider) {


We'll need it - open-telemetry/opentelemetry-java#3807

…ntation name

…nstrumentation into log4j2-otel-appender

trask

thx @jack-berg!

jack-berg · 2021-11-29T15:36:29Z

Can this PR be merged?

trask · 2021-11-29T17:46:28Z

🎉

* WIP * Use Log SDK snapshot * Update to 1.9.0 release, refine * Remove appenders from common log config * Respond to PR feedback * Update readme * Switch to compileOnly log sdk dependency, use logger name as instrumentation name * Switch to minimalist LogEvent mapping * PR feedback * PR feedback

WIP

275f467

jack-berg mentioned this pull request Oct 13, 2021

Prototype Log4j2 Appender #4374

Closed

jack-berg commented Oct 13, 2021

View reviewed changes

This was referenced Oct 14, 2021

LogData nullable spanId, traceId open-telemetry/opentelemetry-java#3745

Merged

Add SimpleLogProcessor open-telemetry/opentelemetry-java#3749

Merged

Refactor Log SDK to implement OTEP-0150 open-telemetry/opentelemetry-java#3759

Merged

jack-berg added 3 commits November 2, 2021 14:05

Merge branch 'main' of github.com:open-telemetry/opentelemetry-java-i…

c99b249

…nstrumentation into log4j2-otel-appender

Use Log SDK snapshot

7111f9a

Update to 1.9.0 release, refine

ecd382d

jack-berg marked this pull request as ready for review November 12, 2021 21:31

jack-berg requested review from anuraaga, iNikem, laurit, mateuszrzeszutek, pavolloffay, trask and tylerbenson as code owners November 12, 2021 21:31

Remove appenders from common log config

45210d2

iNikem reviewed Nov 13, 2021

View reviewed changes

mateuszrzeszutek reviewed Nov 13, 2021

View reviewed changes

anuraaga reviewed Nov 15, 2021

View reviewed changes

jack-berg added 4 commits November 15, 2021 12:15

Respond to PR feedback

18a433a

Update readme

819b806

Switch to compileOnly log sdk dependency, use logger name as instrume…

e0f65f6

…ntation name

Switch to minimalist LogEvent mapping

d3d512b

anuraaga reviewed Nov 16, 2021

View reviewed changes

Comment thread instrumentation/log4j/log4j-2.13.2/library/build.gradle.kts Outdated

Comment thread ...3.2/library/src/main/java/io/opentelemetry/instrumentation/log4j/v2_13_2/LogEventMapper.java Outdated

jack-berg added 2 commits November 16, 2021 18:10

Merge branch 'main' of github.com:open-telemetry/opentelemetry-java-i…

79cb6d9

…nstrumentation into log4j2-otel-appender

PR feedback

db82669

anuraaga approved these changes Nov 17, 2021

View reviewed changes

mateuszrzeszutek approved these changes Nov 17, 2021

View reviewed changes

trask approved these changes Nov 17, 2021

View reviewed changes

PR feedback

d730612

trask merged commit 5c36ed3 into open-telemetry:main Nov 29, 2021

anuraaga mentioned this pull request Dec 1, 2021

Let LogDataBuilder take a SpanContext directly. open-telemetry/opentelemetry-java#3942

Merged

	builder.setSeverityText(logEvent.getLevel().name());
	builder.setSeverityText(level.name());

Uh oh!

Conversation

jack-berg commented Oct 13, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jack-berg commented Nov 12, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jack-berg Nov 16, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

anuraaga left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jack-berg Nov 15, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jack-berg commented Oct 13, 2021 •

edited

Loading

jack-berg Nov 16, 2021 •

edited

Loading

anuraaga left a comment •

edited

Loading

jack-berg Nov 15, 2021 •

edited

Loading