Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion src/vizier/services/agent/kelvin/kelvin_main.cc
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@
#include "src/common/base/base.h"
#include "src/common/event/nats.h"
#include "src/common/signal/signal.h"
#include "src/common/system/kernel_version.h"
Comment thread
benkilimnik marked this conversation as resolved.
#include "src/shared/version/version.h"

DEFINE_string(nats_url, gflags::StringFromEnv("PL_NATS_URL", "pl-nats"),
Expand Down Expand Up @@ -88,8 +89,11 @@ int main(int argc, char** argv) {
std::string mds_addr =
absl::Substitute("$0.$1.svc:$2", FLAGS_mds_addr, FLAGS_namespace, FLAGS_mds_port);

px::system::KernelVersion kernel_version = px::system::GetCachedKernelVersion();
LOG(INFO) << absl::Substitute("Pixie Kelvin. Kernel version: $0", kernel_version.ToString());
Comment thread
benkilimnik marked this conversation as resolved.
Outdated

auto manager = KelvinManager::Create(agent_id, FLAGS_pod_name, FLAGS_host_ip, addr,
FLAGS_rpc_port, FLAGS_nats_url, mds_addr)
FLAGS_rpc_port, FLAGS_nats_url, mds_addr, kernel_version)
.ConsumeValueOrDie();

TerminationHandler::set_manager(manager.get());
Expand Down
7 changes: 5 additions & 2 deletions src/vizier/services/agent/kelvin/kelvin_manager.h
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@
#include <string>
#include <utility>

#include "src/common/system/kernel_version.h"
#include "src/vizier/services/agent/shared/manager/manager.h"

namespace px {
Expand All @@ -43,9 +44,11 @@ class KelvinManager : public Manager {
KelvinManager() = delete;
KelvinManager(sole::uuid agent_id, std::string_view pod_name, std::string_view host_ip,
std::string_view addr, int grpc_server_port, std::string_view nats_url,
std::string_view mds_url)
std::string_view mds_url, px::system::KernelVersion kernel_version)
// TODO(@benkilimnik): Kernel version may not be needed in kelvin, only in PEMManager for
// script selection based on HostInfo. Could use dummy value or refactor Manager class.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed that Kelvin may not need kernel version, at least for the purposes we have right now. Is it possible to remove from Kelvin (or is there some common base class where we add kernel version as an arg. making it required as an arg. here)?

@benkilimnik benkilimnik Sep 6, 2023

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is a common base class Manager in vizier/services/agent/shared/manager/manager.{h,cc} to which I've added the kernel version.

I think we could handle this in a number of ways:

  1. Set a dummy value for the kernel version in KelvinManager.

  2. Set a flag in Manager

protected:
    Manager(..., bool use_kernel_version = true);
Manager::Manager(..., bool use_kernel_version) {
    if (use_kernel_version) {
        // Find and set the kernel version
    }
    // other initialization
}
KelvinManager::KelvinManager(...) 
    : Manager(..., false) { // Pass false to indicate that we don't need to go find the kernel version
    // KelvinManager specific initialization
}
  1. Refactor Manager such that it does not require the kernel version in its constructor, but provides a separate method to set it. Derived classes like PEMManager set the kernel version after construction if needed.

  2. Make the kernel version optional in the base Manager class e.g. std::optional<px::system::KernelVersion>

  3. Keep kernel version in Kelvin

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have another proposal for addressing this. What if we mutated the info_ member's kernel_version field within the PEMManager's constructor body? This would allow us to remove any references of kernel version from the Manager base class and the KelvinManager.

This commit (1795988) shows an example of what I mean. If we like that direction, I think we should also add tests to ensure that a KelvinManager and PEMManager have an info_ member in the correct state after construction: former having kernel_version uninitialized and the latter having it set to a kernel version.

I see this as an improvement to option 3 because it ensures that a newly minted PEMManager is properly configured and reduces risk of using the API incorrectly (forgetting to call the kernel version setter).

@benkilimnik benkilimnik Sep 7, 2023

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That seems like a more robust solution.

Regarding testing, are you imagining something like this, where we check that the kernel version is initialized after construction? (I think the member function info() is protected, so we'd need a getter or friend test function).

class PEMManagerTest : public ::testing::Test {
    protected:
        PEMManagerTest() {
            agent_info_ = agent::Info{};
            agent_info_.agent_id = sole::uuid4();
            agent_info_.hostname = "hostname";
            agent_info_.address = "address";
            agent_info_.pod_name = "pod_name";
            agent_info_.host_ip = "host_ip";
            agent_info_.kernel_version =
                system::ParseKernelVersionString("5.15.0-106-generic").ValueOrDie();
        }

    agent::Info agent_info_;
};

TEST_F(PEMManagerTest, ConstructorTest) {
    auto manager =
      PEMManager::Create(agent_info_.agent_id, agent_info_.pod_name, agent_info_.host_ip,
                         "nats_url", agent_info_.kernel_version)
          .ConsumeValueOrDie();  // Remove the semicolon before .ConsumeValueOrDie()
    EXPECT_EQ(manager->info()->agent_id, agent_info_.agent_id);
    EXPECT_EQ(manager->info()->pod_name, agent_info_.pod_name);
    EXPECT_EQ(manager->info()->host_ip, agent_info_.host_ip);
    EXPECT_EQ(manager->info()->kernel_version, agent_info_.kernel_version);
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@benkilimnik that's what I was thinking and I think making it a friend test function sounds good.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because of the way PEMManager inherits from Manager I have actually found it rather difficult to write a test case for this. When I instantiate the PEMManager I get this error

F20230908 07:05:26.144451    12 statusor.h:148] Check failed: _s.ok() Bad Status: Internal : Failed to read file ../../services/certs/ca.crt (No such file or directory)

which I believe comes from SSL::DefaultGRPCClientCreds() in manager.cc:103

Manager::Manager(sole::uuid agent_id, std::string_view pod_name, std::string_view host_ip,
                 int grpc_server_port, services::shared::agent::AgentCapabilities capabilities,
                 services::shared::agent::AgentParameters parameters, std::string_view nats_url,
                 std::string_view mds_url)
    : grpc_channel_creds_(SSL::DefaultGRPCClientCreds()),
...

I think a test that mocks out parts of the Manager constructor will require some design changes. As I see it, we could:

  1. Move initialization logic out of the Manager constructor, so that we can avoid calling it in tests and only instantiate the object
  2. Instead of directly creating instances / reading files inside the constructor, we could pass them as parameters, so that we can pass mock versions in tests.
  3. Instead of inheriting from Manager, we could make PEMManager and Manager implement the same interface. Then, PEMManager can have a Manager member that it delegates to for the shared functionality. This way, we could mock out the entire Manager when testing PEMManager.

@ddelnano @etep eager to hear your thoughts on this.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After some discussion with @ddelnano, I ended up adding a private constructor to PEMManager that contains a flag to skip the Init() of the base Manager. This flag is set to false in the public constructor and true in friend tests which call it.
The cert error was fixed by setting FLAGS_disable_SSL = true;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd be ok with option (5) keep the version in Kelvin if it nets out to "simpler." It appears we already have a "way" to avoid going here, but is the pain worth it? I am ok with either route btw.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Despite advocating for the current direction, I do think that having the kernel version for both is likely simpler.

Sorry for causing additional change @benkilimnik, but after reconsidering (5) is probably best. If the Manager class was an area where future test coverage would helpful that might tip the scale, but as we've seen it's very difficult to exercise its code.

: Manager(agent_id, pod_name, host_ip, grpc_server_port, KelvinManager::Capabilities(),
KelvinManager::Parameters(), nats_url, mds_url) {
KelvinManager::Parameters(), nats_url, mds_url, kernel_version) {
info()->address = std::string(addr);
}

Expand Down
1 change: 1 addition & 0 deletions src/vizier/services/agent/pem/BUILD.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,7 @@ pl_cc_binary(
deps = [
":cc_library",
"//src/common/signal:cc_library",
"//src/common/system:cc_library",
"//src/shared/version:cc_library",
"//src/shared/version:version_linkstamp",
"//src/vizier/services/agent/shared/base:cc_library",
Expand Down
8 changes: 6 additions & 2 deletions src/vizier/services/agent/pem/pem_main.cc
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@

#include "src/common/base/base.h"
#include "src/common/signal/signal.h"
#include "src/common/system/kernel_version.h"
#include "src/shared/version/version.h"

DEFINE_string(nats_url, gflags::StringFromEnv("PL_NATS_URL", "pl-nats"),
Expand Down Expand Up @@ -65,8 +66,11 @@ int main(int argc, char** argv) {
if (FLAGS_host_ip.length() == 0) {
LOG(FATAL) << "The HOST_IP must be specified";
}
auto manager = PEMManager::Create(agent_id, FLAGS_pod_name, FLAGS_host_ip, FLAGS_nats_url)
.ConsumeValueOrDie();
px::system::KernelVersion kernel_version = px::system::GetCachedKernelVersion();
LOG(INFO) << absl::Substitute("Pixie PEM. Kernel version: $0", kernel_version.ToString());
Comment thread
benkilimnik marked this conversation as resolved.
Outdated
auto manager =
PEMManager::Create(agent_id, FLAGS_pod_name, FLAGS_host_ip, FLAGS_nats_url, kernel_version)
.ConsumeValueOrDie();

TerminationHandler::set_manager(manager.get());

Expand Down
12 changes: 8 additions & 4 deletions src/vizier/services/agent/pem/pem_manager.h
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@

#include <prometheus/gauge.h>

#include "src/common/system/kernel_version.h"
#include "src/stirling/stirling.h"
#include "src/vizier/services/agent/pem/tracepoint_manager.h"
#include "src/vizier/services/agent/shared/manager/manager.h"
Expand All @@ -52,15 +53,18 @@ class PEMManager : public Manager {
protected:
PEMManager() = delete;
PEMManager(sole::uuid agent_id, std::string_view pod_name, std::string_view host_ip,
std::string_view nats_url)
std::string_view nats_url, px::system::KernelVersion kernel_version)
: PEMManager(agent_id, pod_name, host_ip, nats_url,
px::stirling::Stirling::Create(px::stirling::CreateSourceRegistryFromFlag())) {}
px::stirling::Stirling::Create(px::stirling::CreateSourceRegistryFromFlag()),
kernel_version) {}

// Constructor which creates the HostInfo for an agent (runs once per node).
PEMManager(sole::uuid agent_id, std::string_view pod_name, std::string_view host_ip,
std::string_view nats_url, std::unique_ptr<stirling::Stirling> stirling)
std::string_view nats_url, std::unique_ptr<stirling::Stirling> stirling,
px::system::KernelVersion kernel_version)
: Manager(agent_id, pod_name, host_ip, /*grpc_server_port*/ 0, PEMManager::Capabilities(),
PEMManager::Parameters(), nats_url,
/*mds_url*/ ""),
/*mds_url*/ "", kernel_version),
stirling_(std::move(stirling)),
node_available_memory_(prometheus::BuildGauge()
.Name("node_available_memory")
Expand Down
2 changes: 2 additions & 0 deletions src/vizier/services/agent/shared/base/info.h
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@

#include <sole.hpp>

#include "src/common/system/kernel_version.h"
#include "src/vizier/services/shared/agentpb/agent.pb.h"

namespace px {
Expand All @@ -42,6 +43,7 @@ struct Info {
std::string address;
std::string pod_name;
std::string host_ip;
system::KernelVersion kernel_version;
services::shared::agent::AgentCapabilities capabilities;
services::shared::agent::AgentParameters parameters;
};
Expand Down
1 change: 1 addition & 0 deletions src/vizier/services/agent/shared/manager/BUILD.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ pl_cc_library(
"//src/carnot",
"//src/common/event:cc_library",
"//src/common/metrics:cc_library",
"//src/common/system:cc_library",
"//src/common/uuid:cc_library",
"//src/shared/metadata:cc_library",
"//src/shared/schema:cc_library",
Expand Down
4 changes: 3 additions & 1 deletion src/vizier/services/agent/shared/manager/manager.cc
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@
#include "src/common/base/base.h"
#include "src/common/metrics/metrics.h"
#include "src/common/perf/perf.h"
#include "src/common/system/kernel_version.h"
Comment thread
benkilimnik marked this conversation as resolved.
Outdated
#include "src/vizier/funcs/context/vizier_context.h"
#include "src/vizier/funcs/funcs.h"
#include "src/vizier/services/agent/shared/manager/chan_cache.h"
Expand Down Expand Up @@ -98,7 +99,7 @@ std::shared_ptr<services::metadata::CronScriptStoreService::Stub> CreateCronScri
Manager::Manager(sole::uuid agent_id, std::string_view pod_name, std::string_view host_ip,
int grpc_server_port, services::shared::agent::AgentCapabilities capabilities,
services::shared::agent::AgentParameters parameters, std::string_view nats_url,
std::string_view mds_url)
std::string_view mds_url, system::KernelVersion kernel_version)
: grpc_channel_creds_(SSL::DefaultGRPCClientCreds()),
time_system_(std::make_unique<px::event::RealTimeSystem>()),
api_(std::make_unique<px::event::APIImpl>(time_system_.get())),
Expand Down Expand Up @@ -134,6 +135,7 @@ Manager::Manager(sole::uuid agent_id, std::string_view pod_name, std::string_vie
info_.parameters = std::move(parameters);
info_.pod_name = std::string(pod_name);
info_.host_ip = std::string(host_ip);
info_.kernel_version = system::KernelVersion(kernel_version);
}

Status Manager::Init() {
Expand Down
3 changes: 2 additions & 1 deletion src/vizier/services/agent/shared/manager/manager.h
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@
#include "src/common/event/event.h"
#include "src/common/event/nats.h"
#include "src/common/metrics/memory_metrics.h"
#include "src/common/system/kernel_version.h"
#include "src/common/uuid/uuid.h"
#include "src/shared/metadata/metadata.h"
#include "src/vizier/funcs/context/vizier_context.h"
Expand Down Expand Up @@ -108,7 +109,7 @@ class Manager : public BaseManager {
Manager(sole::uuid agent_id, std::string_view pod_name, std::string_view host_ip,
int grpc_server_port, services::shared::agent::AgentCapabilities capabilities,
services::shared::agent::AgentParameters parameters, std::string_view nats_url,
std::string_view mds_url);
std::string_view mds_url, px::system::KernelVersion kernel_version);
Status Init();

Status RegisterMessageHandler(MsgCase c, std::shared_ptr<MessageHandler> handler,
Expand Down
14 changes: 14 additions & 0 deletions src/vizier/services/agent/shared/manager/registration.cc
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,10 @@
#include <utility>
#include <vector>

#include "src/vizier/services/agent/shared/base/info.h"
#include "src/vizier/services/agent/shared/manager/manager.h"
#include "src/vizier/services/agent/shared/manager/registration.h"
#include "src/vizier/services/metadata/metadatapb/service.grpc.pb.h"
Comment thread
benkilimnik marked this conversation as resolved.
Outdated

namespace px {
namespace vizier {
Expand Down Expand Up @@ -56,6 +58,16 @@ RegistrationHandler::RegistrationHandler(px::event::Dispatcher* dispatcher, Info
});
}

// convert px::system::KernelVersion to px::vizier::services::shared::agent:KernelVersion
::px::vizier::services::shared::agent::KernelVersion KernelToProto(
const system::KernelVersion& kv) {
::px::vizier::services::shared::agent::KernelVersion kv_proto;
kv_proto.set_version(kv.version);
kv_proto.set_major_rev(kv.major_rev);
kv_proto.set_minor_rev(kv.minor_rev);
return kv_proto;
}

Status RegistrationHandler::DispatchRegistration() {
// Send the registration request.
messages::VizierMessage req;
Expand All @@ -79,6 +91,8 @@ Status RegistrationHandler::DispatchRegistration() {
host_info->set_hostname(agent_info()->hostname);
host_info->set_pod_name(agent_info()->pod_name);
host_info->set_host_ip(agent_info()->host_ip);
auto kernel_version_proto = KernelToProto(agent_info()->kernel_version);
host_info->mutable_kernel()->CopyFrom(kernel_version_proto);
*req_info->mutable_capabilities() = agent_info()->capabilities;
*req_info->mutable_parameters() = agent_info()->parameters;

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,8 @@ class RegistrationHandlerTest : public ::testing::Test {
agent_info_.pod_name = "pod_name";
agent_info_.host_ip = "host_ip";
agent_info_.capabilities.set_collects_data(true);
agent_info_.kernel_version =
system::ParseKernelVersionString("5.15.0-106-generic").ValueOrDie();

auto register_hook = [this](uint32_t asid) -> Status {
called_register_++;
Expand Down Expand Up @@ -112,6 +114,7 @@ TEST_F(RegistrationHandlerTest, RegisterAgent) {
EXPECT_EQ(agent_info_.hostname, req.info().host_info().hostname());
EXPECT_EQ(agent_info_.pod_name, req.info().host_info().pod_name());
EXPECT_EQ(agent_info_.host_ip, req.info().host_info().host_ip());
EXPECT_EQ(agent_info_.kernel_version.version, req.info().host_info().kernel().version());

auto registration_ack = std::make_unique<messages::VizierMessage>();
registration_ack->mutable_register_agent_response()->set_asid(10);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -118,6 +118,11 @@ func TestRegisterAgent(t *testing.T) {
HostInfo: &agentpb.HostInfo{
Hostname: "localhost",
HostIP: "127.0.0.4",
Kernel: &agentpb.KernelVersion{
Version: 5,
MajorRev: 19,
MinorRev: 0,
},
},
AgentID: upb,
Capabilities: &agentpb.AgentCapabilities{
Expand Down
Loading